From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id IIoZD9tDgWKQFQEAbAwnHQ (envelope-from ) for ; Sun, 15 May 2022 20:18:03 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id KDYLD9tDgWIVWQAA9RJhRA (envelope-from ) for ; Sun, 15 May 2022 20:18:03 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D2C1127EC4 for ; Sun, 15 May 2022 20:18:02 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 662A25F771; Sun, 15 May 2022 18:17:26 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id 883BC5F766 for ; Sun, 15 May 2022 18:17:24 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id 578395FC42; Sun, 15 May 2022 14:17:23 -0400 (EDT) Received: (nullmailer pid 57848 invoked by uid 1000); Sun, 15 May 2022 18:17:13 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH 16/17] CLI/git: add safety checks for checkout and commit Date: Sun, 15 May 2022 15:14:21 -0300 Message-Id: <20220515181421.57088-17-david@tethera.net> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220515181421.57088-1-david@tethera.net> References: <20220515181421.57088-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: T542PCU5NSZHPNRR4WAO3RKUEDCOYGXG X-Message-ID-Hash: T542PCU5NSZHPNRR4WAO3RKUEDCOYGXG X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1652638682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=xtmAAUPGAWiNobR7HSYO7DH657CrOzZDup8YnqyFziM=; b=jtDXblIZOp7Y565q8L9Ku8TI6x05aj8pNX/lvoxU8Vac2ZWvahWZh4c+rDcFrhj57WH5GJ lDkxCDd34xqK7y+YO6uuEk2gxOyFNfeFlew/VGseM5WPmD5j78i0MRu5Hab1AhsGId6pZW n/B0YCNIGADYhaeAJqhMUmztUCgd3Ow2MdNHd0o3AQYCluGTJW3AKGxt3i5kPOQ/2Lz4qO /Le0AAnI4EMT8w31eC08138GQ3p02og1S8zSQC/mikn1e6KN/+JAAsf5sK51uKWsw+8l3P XrzOOPQktALoiQf5hrwmXrcti07M7U3aiIBp/sTl72WFCP76W6hyzM+4va8EAA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1652638682; a=rsa-sha256; cv=none; b=jh1UhEM4l0Aqag8zeYvvZZVTCS/C4GXgCukWPZZj4fgx0/tzX+HHe2VRFdXT84p2p6Gmsn r6kkB/sFnTGudcyD7BRExU8Vq3rRI9GjhTD2yr/Bw4Y+SAVCxpffBlRT0kuPdnenR5dM3i TdVxLh7ZsodWRn9dmdhQgwPt3Nmd4xnYrbht8AElFY0C4KvCWq9OOfLMkMGFCts3CyQqJT Uk2ccr095Txc3M0veODBkBdJIhFrIh95ggVGNErmsxqXaALftcW32WT4iOqHR+Y5v+xK3R s5s45Hn9yzI5tBfhYuAxSO4/FZrXbyGxG0qychPsFEfHid/Vu834I0RDliMPoQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.45 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: D2C1127EC4 X-Spam-Score: -1.45 X-Migadu-Scanner: scn0.migadu.com X-TUID: t9U+/DJ61KY0 Commits or checkouts that modify a large fraction of the messages in the database should be relatively rare (and in some automated process, probably non-existent). For initial setup, where such operations are expected, the user can pass --force. --- doc/man1/notmuch-config.rst | 7 ++++++ doc/man1/notmuch-git.rst | 14 ++++++++++-- notmuch-git.in | 43 +++++++++++++++++++++++++++++++++++-- test/T850-git.sh | 39 ++++++++++++++++++++++++++++----- 4 files changed, 94 insertions(+), 9 deletions(-) diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst index e2e9a632..388315f6 100644 --- a/doc/man1/notmuch-config.rst +++ b/doc/man1/notmuch-config.rst @@ -111,6 +111,13 @@ paths are presumed relative to `$HOME` for items in section Default location for git repository for :any:`notmuch-git`. +.. nmconfig:: git.safe_fraction + + Some :any:`notmuch-git` operations check that the fraction of + messages changed (in the database or in git, as appropriate) is not + too large. This item controls what fraction of total messages is + considered "not too large". + .. nmconfig:: git.tag_prefix Default tag prefix (filter) for :any:`notmuch-git`. diff --git a/doc/man1/notmuch-git.rst b/doc/man1/notmuch-git.rst index ad859b80..fa7a748e 100644 --- a/doc/man1/notmuch-git.rst +++ b/doc/man1/notmuch-git.rst @@ -73,13 +73,18 @@ Dump a tar archive of a committed tag set using 'git archive'. See :manpage:`git-archive(1)`. Arguments to `git-archive` are reordered so that *tree-ish* comes last. -.. option:: checkout +.. option:: checkout [-f|--force] Update the notmuch database from Git. This is mainly useful to discard your changes in notmuch relative to Git. + .. describe:: [-f|--force] + + Override checks that prevent modifying tags for large fractions of + messages in the database. See also :nmconfig:`git.safe_fraction`. + .. option:: clone Create a local `notmuch git` repository from a remote source. @@ -94,7 +99,7 @@ upstreams. section of :manpage:`git-clone(1)` for more information on specifying repositories. -.. option:: commit [message] +.. option:: commit [-f|--force] [message] Commit prefix-matching tags from the notmuch database to Git. @@ -102,6 +107,11 @@ Commit prefix-matching tags from the notmuch database to Git. Optional text for the commit message. + .. describe:: -f|--force + + Override checks that prevent modifying tags for large fractions of + messages in the database. See also :nmconfig:`git.safe_fraction`. + .. option:: fetch [remote] Fetch changes from the remote repository. diff --git a/notmuch-git.in b/notmuch-git.in index 3e5205e8..6505c2e5 100755 --- a/notmuch-git.in +++ b/notmuch-git.in @@ -241,6 +241,16 @@ def _tag_query(prefix=None): prefix = TAG_PREFIX return '(tag (starts-with "{:s}"))'.format(prefix.replace('"','\\\"')) +def count_messages(prefix=None): + "count messages with a given prefix." + (status, stdout, stderr) = _spawn( + args=['notmuch', 'count', '--query=sexp', _tag_query(prefix)], + stdout=_subprocess.PIPE, wait=True) + if status != 0: + _LOG.error("failed to run notmuch config") + sys.exit(1) + return int(stdout.rstrip()) + def get_tags(prefix=None): "Get a list of tags with a given prefix." (status, stdout, stderr) = _spawn( @@ -359,7 +369,22 @@ class CachedIndex: _git(args=['read-tree', self.current_treeish], wait=True) -def commit(treeish='HEAD', message=None): +def check_safe_fraction(status): + safe = 0.1 + conf = _notmuch_config_get ('git.safe_fraction') + if conf and conf != '': + safe=float(conf) + + total = count_messages (TAG_PREFIX) + change = len(status['added'])+len(status['deleted'])+len(status['missing']) + fraction = change/total + _LOG.debug('total messages {:d}, change: {:d}, fraction: {:f}'.format(total,change,fraction)) + if fraction > safe: + _LOG.error('safe fraction {:f} exceeded, stopping.'.format(safe)) + _LOG.error('Use --force to override or reconfigure git.safe_fraction.') + exit(1) + +def commit(treeish='HEAD', message=None, force=False): """ Commit prefix-matching tags from the notmuch database to Git. """ @@ -370,6 +395,9 @@ def commit(treeish='HEAD', message=None): _LOG.warning('Nothing to commit') return + if not force: + check_safe_fraction (status) + with CachedIndex(NOTMUCH_GIT_DIR, treeish) as index: try: _update_index(status=status) @@ -447,7 +475,7 @@ def init(remote=None): wait=True) -def checkout(): +def checkout(force=None): """ Update the notmuch database from Git. @@ -455,6 +483,10 @@ def checkout(): to Git. """ status = get_status() + + if not force: + check_safe_fraction(status) + with _spawn( args=['notmuch', 'tag', '--batch'], stdin=_subprocess.PIPE) as p: for id, tags in status['added'].items(): @@ -948,6 +980,10 @@ if __name__ == '__main__': help=( "Argument passed through to 'git archive'. Set anything " 'before , see git-archive(1) for details.')) + elif command == 'checkout': + subparser.add_argument( + '-f', '--force', action='store_true', + help='checkout a large fraction of tags.') elif command == 'clone': subparser.add_argument( 'repository', @@ -956,6 +992,9 @@ if __name__ == '__main__': 'URLS section of git-clone(1) for more information on ' 'specifying repositories.')) elif command == 'commit': + subparser.add_argument( + '-f', '--force', action='store_true', + help='commit a large fraction of tags.') subparser.add_argument( 'message', metavar='MESSAGE', default='', nargs='?', help='Text for the commit message.') diff --git a/test/T850-git.sh b/test/T850-git.sh index 8f91b612..508615e1 100755 --- a/test/T850-git.sh +++ b/test/T850-git.sh @@ -25,12 +25,25 @@ test_expect_equal "$output" "true" test_begin_subtest "clone" test_expect_success "notmuch git -p '' -C tags.git clone remote.git" +test_begin_subtest "initial commit needs force" +test_expect_code 1 "notmuch git -C tags.git commit" + test_begin_subtest "commit" -notmuch git -C tags.git commit +notmuch git -C tags.git commit --force git -C tags.git ls-tree -r --name-only HEAD | xargs dirname | sort -u | sed s,tags/,id:, > OUTPUT notmuch search --output=messages '*' | sort > EXPECTED test_expect_equal_file_nonempty EXPECTED OUTPUT +test_begin_subtest "commit --force succeeds" +notmuch git -C force.git init +test_expect_success "notmuch git -C force.git commit --force" + +test_begin_subtest "changing git.safe_fraction succeeds" +notmuch config set git.safe_fraction 1 +notmuch git -C force2.git init +test_expect_success "notmuch git -C force2.git commit" +notmuch config set git.safe_fraction + test_begin_subtest "commit, with quoted tag" notmuch git -C clone2.git clone tags.git git -C clone2.git ls-tree -r --name-only HEAD | grep /inbox > BEFORE @@ -64,12 +77,12 @@ test_expect_equal_file_nonempty EXPECTED OUTPUT test_begin_subtest "commit (change prefix)" notmuch tag +test::one id:20091117190054.GU3165@dottiness.seas.harvard.edu -notmuch git -C tags.git -p 'test::' commit +notmuch git -C tags.git -p 'test::' commit --force git -C tags.git ls-tree -r --name-only HEAD | grep 20091117190054 | sort > OUTPUT echo "--------------------------------------------------" >> OUTPUT notmuch tag -test::one id:20091117190054.GU3165@dottiness.seas.harvard.edu -notmuch git -C tags.git commit +notmuch git -C tags.git commit --force git -C tags.git ls-tree -r --name-only HEAD | grep 20091117190054 | sort >> OUTPUT cat < EXPECTED @@ -81,10 +94,26 @@ tags/20091117190054.GU3165@dottiness.seas.harvard.edu/unread EOF test_expect_equal_file_nonempty EXPECTED OUTPUT +backup_database +test_begin_subtest "large checkout needs --force" +notmuch tag -inbox '*' +test_expect_code 1 "notmuch git -C tags.git checkout" +restore_database + +test_begin_subtest "checkout (git.safe_fraction)" +notmuch git -C force3.git clone tags.git +notmuch dump > BEFORE +notmuch tag -inbox '*' +notmuch config set git.safe_fraction 1 +notmuch git -C force3.git checkout +notmuch config set git.safe_fraction +notmuch dump > AFTER +test_expect_equal_file_nonempty BEFORE AFTER + test_begin_subtest "checkout" notmuch dump > BEFORE notmuch tag -inbox '*' -notmuch git -C tags.git checkout +notmuch git -C tags.git checkout --force notmuch dump > AFTER test_expect_equal_file_nonempty BEFORE AFTER @@ -111,7 +140,7 @@ test_expect_equal_file EXPECTED OUTPUT test_begin_subtest "fetch" notmuch tag +test2 id:20091117190054.GU3165@dottiness.seas.harvard.edu -notmuch git -C remote.git commit +notmuch git -C remote.git commit --force notmuch tag -test2 id:20091117190054.GU3165@dottiness.seas.harvard.edu notmuch git -C tags.git fetch notmuch git -C tags.git status > OUTPUT -- 2.35.2