From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id yJ7KBT3jE2NZEQEAbAwnHQ (envelope-from ) for ; Sun, 04 Sep 2022 01:29:01 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id kE3gBT3jE2Or5QAA9RJhRA (envelope-from ) for ; Sun, 04 Sep 2022 01:29:01 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C5B3931CA4 for ; Sun, 4 Sep 2022 01:29:00 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 18DD85F3FE; Sat, 3 Sep 2022 23:28:54 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by mail.notmuchmail.org (Postfix) with ESMTP id A62BF5F38D for ; Sat, 3 Sep 2022 23:28:50 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id E8F965FBC2; Sat, 3 Sep 2022 19:28:49 -0400 (EDT) Received: (nullmailer pid 1474100 invoked by uid 1000); Sat, 03 Sep 2022 23:28:47 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH 3/3] lib: index attachements with mime types matching index.as_text Date: Sat, 3 Sep 2022 20:28:39 -0300 Message-Id: <20220903232839.1473915-4-david@tethera.net> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220903232839.1473915-1-david@tethera.net> References: <20220903232839.1473915-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: OWZMJG2YDVGPGHVCFGG54323OCGWS7BV X-Message-ID-Hash: OWZMJG2YDVGPGHVCFGG54323OCGWS7BV X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1662247740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=5tTOyuNxW8yUNxqyCgX9xDSidrW2HkqCNYyAkp6rkNw=; b=DnUIIywkFhO2huVQYDUsdGr7qBOb3nUrCn0bvsupSU0KhrzZBt+CxFmktrIDQkT/X+AMLH 4BKtwsXpgpfCG+s+eedgoKa0OUhPGGknAjmjveVbRaZWoFZgGQNMt4UIjQCwSPDtoWhl76 UbAOiqZ8+znX/BAus7cECEhczCqJRtwWK2r0OpuGQ+MhH78SyBhJxRGcaNfNfnMfWhfi8K EfgzX6wNLrfbozwxe1HGWC2v/Xg1jH9bhmH1qeQC8gW5MDRag2AISWr2RJHpahvSCRi+bn 2090Iim9jQY5x0FBI6tczeNOfhNGPHLSfnLfIhxzgU3yL9ruwEvEr5aluV8qmQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1662247740; a=rsa-sha256; cv=none; b=VV30OR9MPm0R+FIud6SK5wEKYP/t58GiTtkg9sBtGvJIAXiRVJnq0Uq0x6epKattJYYVWG TgPrzTpIuNhh55VpwpjjrSbsNYWqb3MsiJ8LL1GeBy7lJhmHJD9F42IyPJXQzMIu5M2SkX FMdroRaVezeBHvGNgZi/BoBWmnTs2BN6f/vJ4OOYUsTmAHl0OlgLl9xUssLdenBNpPfAHZ dmexxancHSPaX9Spb995fyi/qvJYNITvdzxySRY5dXQNOjJDbmen0yCrC7HltnKufNXuCR Kb+QG1fzckKjt6dKnhTIXkPHch2TlzdH35/L8axOu94uPzkb83pKcLURZoZtCQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.42 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: C5B3931CA4 X-Spam-Score: -1.42 X-Migadu-Scanner: scn1.migadu.com X-TUID: g+VBk2gQTtW5 Instead of skipping indexing all attachments, we check of a (user configured) mime type that is indexable as text. --- doc/man1/notmuch-config.rst | 7 +++++++ lib/database.cc | 12 ++++++++++++ lib/index.cc | 25 ++++++++++++++++++++++--- lib/notmuch-private.h | 4 ++++ test/T050-new.sh | 37 ++++++++++++++++++++++++++++++++++++- 5 files changed, 81 insertions(+), 4 deletions(-) diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst index 388315f6..caa3bd65 100644 --- a/doc/man1/notmuch-config.rst +++ b/doc/man1/notmuch-config.rst @@ -122,6 +122,13 @@ paths are presumed relative to `$HOME` for items in section Default tag prefix (filter) for :any:`notmuch-git`. +.. nmconfig:: index.as_text + + List of regular expressions (without delimiters) for MIME types to + be indexed as text. Currently this applies only to attachments. + + History: This configuration value was introduced in notmuch 0.38. + .. nmconfig:: index.decrypt Policy for decrypting encrypted messages during indexing. Must be diff --git a/lib/database.cc b/lib/database.cc index c05d70d3..6b962a15 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -1573,3 +1573,15 @@ notmuch_database_status_string (const notmuch_database_t *notmuch) { return notmuch->status_string; } + +bool +_notmuch_database_indexable_as_text (notmuch_database_t *notmuch, const char *mime_string) +{ + for (size_t i = 0; i < notmuch->index_as_text_length; i++) { + if (regexec (¬much->index_as_text[i], mime_string, 0, NULL, 0) == 0) { + return true; + } + } + + return false; +} diff --git a/lib/index.cc b/lib/index.cc index 728bfb22..629dcb22 100644 --- a/lib/index.cc +++ b/lib/index.cc @@ -380,6 +380,23 @@ _index_pkcs7_part (notmuch_message_t *message, GMimeObject *part, _notmuch_message_crypto_t *msg_crypto); +static bool +_indexable_as_text (notmuch_message_t *message, GMimeObject *part) +{ + GMimeContentType *content_type = g_mime_object_get_content_type (part); + notmuch_database_t *notmuch = notmuch_message_get_database (message); + + if (content_type) { + char *mime_string = g_mime_content_type_get_mime_type (content_type); + if (mime_string) { + bool ret = _notmuch_database_indexable_as_text (notmuch, mime_string); + g_free (mime_string); + return ret; + } + } + return false; +} + /* Callback to generate terms for each mime part of a message. */ static void _index_mime_part (notmuch_message_t *message, @@ -497,9 +514,11 @@ _index_mime_part (notmuch_message_t *message, _notmuch_message_add_term (message, "tag", "attachment"); _notmuch_message_gen_terms (message, "attachment", filename); - /* XXX: Would be nice to call out to something here to parse - * the attachment into text and then index that. */ - goto DONE; + if (! _indexable_as_text (message, part)) { + /* XXX: Would be nice to call out to something here to parse + * the attachment into text and then index that. */ + goto DONE; + } } byte_array = g_byte_array_new (); diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h index 1d3d2b0c..c19ee8e2 100644 --- a/lib/notmuch-private.h +++ b/lib/notmuch-private.h @@ -259,6 +259,10 @@ _notmuch_database_filename_to_direntry (void *ctx, notmuch_find_flags_t flags, char **direntry); +bool +_notmuch_database_indexable_as_text (notmuch_database_t *notmuch, + const char *mime_string); + /* directory.cc */ notmuch_directory_t * diff --git a/test/T050-new.sh b/test/T050-new.sh index cb67889c..427c5b22 100755 --- a/test/T050-new.sh +++ b/test/T050-new.sh @@ -455,12 +455,47 @@ Date: Fri, 17 Jun 2016 22:14:41 -0400 EOF test_expect_equal_file EXPECTED OUTPUT +add_email_corpus indexing +test_begin_subtest "index text/* attachments, no config" +messages=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain) +count=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz) +test_expect_equal "$messages,$count" "1,0" + +notmuch config set index.as_text "text/" add_email_corpus indexing test_begin_subtest "index text/* attachments" -test_subtest_known_broken notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT test_expect_equal_file_nonempty EXPECTED OUTPUT +test_begin_subtest "reindex text/* attachments, no config" +notmuch config set index.as_text +notmuch reindex '*' +messages=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain) +count=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz) +test_expect_equal "$messages,$count" "1,0" + +test_begin_subtest "reindex text/* attachments" +notmuch config set index.as_text text/ +notmuch reindex '*' +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "reindex text/* attachments, second regex" +notmuch config set index.as_text "blahblah;text/" +notmuch reindex '*' +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "reindex text/* attachments, bad regex" +notmuch config set index.as_text '[' +notmuch reindex '*' >& OUTPUT +cat< EXPECTED +Error in index.as_text: Invalid regular expression: [ +EOF +test_expect_equal_file EXPECTED OUTPUT + test_done -- 2.35.2