From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 81E936DE0FBC for ; Wed, 4 Dec 2019 10:09:11 -0800 (PST) Authentication-Results: arlo.cworth.org; dkim=permerror (0-bit key) header.d=fifthhorseman.net header.i=@fifthhorseman.net header.b="8DlW7bjX"; dkim=pass (2048-bit key; unprotected) header.d=fifthhorseman.net header.i=@fifthhorseman.net header.b="H0Cki9hF"; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -1.495 X-Spam-Level: X-Spam-Status: No, score=-1.495 tagged_above=-999 required=5 tests=[AWL=1.006, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9srW4TS9RRLA for ; Wed, 4 Dec 2019 10:09:10 -0800 (PST) Received: from che.mayfirst.org (che.mayfirst.org [162.247.75.118]) by arlo.cworth.org (Postfix) with ESMTPS id 3BCF46DE0F74 for ; Wed, 4 Dec 2019 10:09:10 -0800 (PST) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019; t=1575482947; h=from : to : subject : in-reply-to : references : date : message-id : mime-version : content-type : from; bh=TPx/1MLkDmcda5q92+l/f/xrkw96VhlbcDqOGUq/S1A=; b=8DlW7bjXddFXNiM30vzBVSabFZIORgvm2nyC/8Xu3nzrs5MxwUxLAwQV dFInduyigDMG72qka1kUpRarFpifBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fifthhorseman.net; i=@fifthhorseman.net; q=dns/txt; s=2019rsa; t=1575482947; h=from : to : subject : in-reply-to : references : date : message-id : mime-version : content-type : from; bh=TPx/1MLkDmcda5q92+l/f/xrkw96VhlbcDqOGUq/S1A=; b=H0Cki9hFBFVzhXgxxgRAgZqx/Cdrrxgi0DPxLUN2x7wdvIfzk2Xr54X+ db+hsBtSGbRqK5BB2jLLXrVh1G83JDnDUYlO3C89mUEnycblaaQI2Ogup5 pHyIV7LTCMd3DBsRQcBw1uEUZonHCOj0q0Eoxd1BG0wZIIpBbv2vk1/EP6 4KcAuEWuMHBEixOaQt7dWLUdGfgMhx0SxWxS5r9iYzT/MUX7znQL8jz9YD 4nyBG9TOKyqlDjLBxi1ruKcaW7kSqofWkATVshP5OMCje6NOB7fsL6v5N4 o+YPhawmBRETBLtXNIcIkET/1snRAwvHHVIs96tgZpNtkkQVHQQ3xg== Received: from fifthhorseman.net (unknown [38.109.115.130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by che.mayfirst.org (Postfix) with ESMTPSA id DEE2EF9A5; Wed, 4 Dec 2019 13:09:07 -0500 (EST) Received: by fifthhorseman.net (Postfix, from userid 1000) id C823C2030E; Wed, 4 Dec 2019 13:09:03 -0500 (EST) From: Daniel Kahn Gillmor To: Antoine =?utf-8?Q?Beaupr=C3=A9?= , notmuch@notmuchmail.org Subject: Re: compacting the notmuch database through systemd In-Reply-To: <87v9qz93b0.fsf@curie.anarc.at> References: <87v9qz93b0.fsf@curie.anarc.at> Autocrypt: addr=dkg@fifthhorseman.net; prefer-encrypt=mutual; keydata= mDMEXEK/AhYJKwYBBAHaRw8BAQdAr/gSROcn+6m8ijTN0DV9AahoHGafy52RRkhCZVwxhEe0K0Rh bmllbCBLYWhuIEdpbGxtb3IgPGRrZ0BmaWZ0aGhvcnNlbWFuLm5ldD6ImQQTFggAQQIbAQUJA8Jn AAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBMS8Lds4zOlkhevpwvIGkReQOOXGBQJcQsbzAhkB AAoJEPIGkReQOOXG4fkBAO1joRxqAZY57PjdzGieXLpluk9RkWa3ufkt3YUVEpH/AP9c+pgIxtyW +FwMQRjlqljuj8amdN4zuEqaCy4hhz/1DbgzBFxCv4sWCSsGAQQB2kcPAQEHQERSZxSPmgtdw6nN u7uxY7bzb9TnPrGAOp9kClBLRwGfiPUEGBYIACYWIQTEvC3bOMzpZIXr6cLyBpEXkDjlxgUCXEK/ iwIbAgUJAeEzgACBCRDyBpEXkDjlxnYgBBkWCAAdFiEEyQ5tNiAKG5IqFQnndhgZZSmuX/gFAlxC v4sACgkQdhgZZSmuX/iVWgD/fCU4ONzgy8w8UCHGmrmIZfDvdhg512NIBfx+Mz9ls5kA/Rq97vz4 z48MFuBdCuu0W/fVqVjnY7LN5n+CQJwGC0MIA7QA/RyY7Sz2gFIOcrns0RpoHr+3WI+won3xCD8+ sVXSHZvCAP98HCjDnw/b0lGuCR7coTXKLIM44/LFWgXAdZjm1wjODbg4BFxCv50SCisGAQQBl1UB BQEBB0BG4iXnHX/fs35NWKMWQTQoRI7oiAUt0wJHFFJbomxXbAMBCAeIfgQYFggAJhYhBMS8Lds4 zOlkhevpwvIGkReQOOXGBQJcQr+dAhsMBQkB4TOAAAoJEPIGkReQOOXGe/cBAPlek5d9xzcXUn/D kY6jKmxe26CTws3ZkbK6Aa5Ey/qKAP0VuPQSCRxA7RKfcB/XrEphfUFkraL06Xn/xGwJ+D0hCw== Date: Wed, 04 Dec 2019 13:09:03 -0500 Message-ID: <871rtk6k00.fsf@fifthhorseman.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Dec 2019 18:09:11 -0000 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Thanks for raising this, Anarcat! One more advantage that i think you haven't noted yet about regular database compaction: "notmuch compact" tends to get rid of a lot of lingering written data that is no longer referenced. While this isn't robust "secure deletion", it's a lot better than not compacting. see https://trac.xapian.org/ticket/742 for more discussion. Some questions below=E2=80=A6 On Sun 2019-12-01 15:52:19 -0500, Antoine Beaupr=C3=A9 wrote: > Thanks to Bremner, I just realized that notmuch-compact(1) is a thing, > and that thing allows me to compress my notmuch databases by about 50%. do you know why you get the large size/speed gain? do you regularly delete files from your message archive? > So I whipped together two systemd units (attached) that will run that > command every month on my notmuch database. Just drop them in > `~/.config/systemd/user/` and run: > > systemctl --user daemon-reload > systemctl --user enable notmuch-compact.timer > systemctl --user start notmuch-compact.timer ("systemctl --user enable --now notmuch-compact.timer" will suffice for the final two commands on any reasonably modern version of systemd) How long does it take for these the notmuch-compact.service to complete? What happens if this is happening when, say, you put your machine to sleep, or you power it down? While notmuch-compact.service is running, does "notmuch new" or "notmuch insert" work? If not, how do they fail (e.g. blocking indefinitely, returning a comprehensible error message)? Can you read your mail while notmuch-compact.service is running? > Maybe those could be shipped with the Debian package somehow? Not sure > how that works, but I think that's how gpg-agent gets started now, if > you want any inspiration... gpg-agent is socket-activated, which is different from the timer-activation you are proposing here. We could easily ship these systemd user unit files in the notmuch package now that #764678 is resolved. Do you think that the timer should be enabled by default? What should happen if the user hasn't set up notmuch? Maybe we need a ConditionPathExists=3D or something like that on either the .timer or the .service? Do we expect this to run even when the user isn't logged in at all (a background compaction?) it always gets more complex when you think about trying to do it at scale :) > It would be great if notmuch-new ran this on its own, when it > thought that this was "important", somehow like git-gc sometimes runs on > its own. I'm not convinced i like this idea without more profiling and an understanding of what it might cause. I have grown to *really* dislike the highly variable latency and warnings caused by GnuPG's "auto-check-trustdb", for example (especially as the keyring grows larger). > [ notmuch-compact.timer: text/plain ] > [Unit] > Description=3Dcompact the notmuch database systemd timer unit descriptions typically include some mention of the duration. See for example: /lib/systemd/system/systemd-tmpfiles-clean.timer "Daily Cleanup of Temporary Directories" /lib/systemd/system/certbot.timer "Run certbot twice daily" /lib/systemd/system/phpsessionclean.timer "Clean PHP session files every 30 mins" I recommend: Description=3DCompact the notmuch database every month > [ notmuch-compact.service: text/plain ] > [Unit] > Description=3Dcompact the notmuch database The convention is to lead with an upper-case letter: Description=3DCompact the notmuch database OK OK enough with the nit-picking! Happy hacking, --dkg --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHQEARYIAB0WIQTJDm02IAobkioVCed2GBllKa5f+AUCXef2PwAKCRB2GBllKa5f +ExIAQDcjtZDCOFMj3Nr8hUwJ6yK4Oq3o3LgFvA6ZiIXXHraBwD3aL9kUOHyIJKL yVzGa+tgMeIdoDRDkEpLQ+/cxPqkCA== =GLnn -----END PGP SIGNATURE----- --=-=-=--