From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 365716DE160C for ; Wed, 22 Mar 2017 04:23:16 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.005 X-Spam-Level: X-Spam-Status: No, score=-0.005 tagged_above=-999 required=5 tests=[AWL=0.006, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GyjvwnkD1YVl for ; Wed, 22 Mar 2017 04:23:15 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id 8EF596DE15C7 for ; Wed, 22 Mar 2017 04:23:15 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.84_2) (envelope-from ) id 1cqeLX-0000gP-V2; Wed, 22 Mar 2017 07:22:31 -0400 Received: (nullmailer pid 12138 invoked by uid 1000); Wed, 22 Mar 2017 11:23:11 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH 7/7] lib/index: add simple html filter Date: Wed, 22 Mar 2017 08:23:06 -0300 Message-Id: <20170322112306.12060-8-david@tethera.net> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170322112306.12060-1-david@tethera.net> References: <20170322112306.12060-1-david@tethera.net> X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 11:23:16 -0000 Just drop all tags --- lib/index.cc | 21 ++++++++++++++++++++- test/T680-html-indexing.sh | 5 ++++- 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/lib/index.cc b/lib/index.cc index fd66762c..324e6e79 100644 --- a/lib/index.cc +++ b/lib/index.cc @@ -206,6 +206,22 @@ filter_filter_uuencode (GMimeFilter *gmime_filter, char *inbuf, size_t inlen, si } static void +filter_filter_html (GMimeFilter *gmime_filter, char *inbuf, size_t inlen, size_t prespace, + char **outbuf, size_t *outlen, size_t *outprespace) +{ + static const scanner_state_t states[] = { + {0, '<', '<', 1, 0}, + {1, '\'', '\'', 4, 2}, /* scanning for quote or > */ + {1, '"', '"', 5, 3}, + {1, '>', '>', 0, 1}, + {4, '\'', '\'', 1, 4}, /* inside single quotes */ + {5, '"', '"', 1, 5}, /* inside double quotes */ + }; + do_filter(states, 1, + gmime_filter, inbuf, inlen, prespace, outbuf, outlen, outprespace); +} + +static void filter_complete (GMimeFilter *filter, char *inbuf, size_t inlen, size_t prespace, char **outbuf, size_t *outlen, size_t *outprespace) { @@ -252,7 +268,10 @@ notmuch_filter_discard_non_terms_new (GMimeContentType *content_type) filter = (NotmuchFilterDiscardNonTerms *) g_object_newv (type, 0, NULL); filter->state = 0; filter->content_type = content_type; - filter->real_filter = filter_filter_uuencode; + if (g_mime_content_type_is_type (content_type, "text", "html")) + filter->real_filter = filter_filter_html; + else + filter->real_filter = filter_filter_uuencode; return (GMimeFilter *) filter; } diff --git a/test/T680-html-indexing.sh b/test/T680-html-indexing.sh index 5e9cc4cb..74f33708 100755 --- a/test/T680-html-indexing.sh +++ b/test/T680-html-indexing.sh @@ -5,10 +5,13 @@ test_description="indexing of html parts" add_email_corpus html test_begin_subtest 'embedded images should not be indexed' -test_subtest_known_broken notmuch search kwpza7svrgjzqwi8fhb2msggwtxtwgqcxp4wbqr4wjddstqmeqa7 > OUTPUT test_expect_equal_file /dev/null OUTPUT +test_begin_subtest 'ignore > in attribute text' +notmuch search swordfish | notmuch_search_sanitize > OUTPUT +test_expect_equal_file /dev/null OUTPUT + test_begin_subtest 'non tag text should be indexed' notmuch search hunter2 | notmuch_search_sanitize > OUTPUT cat < EXPECTED -- 2.11.0