From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 92E8A429E21 for ; Sun, 6 Nov 2011 13:07:57 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 1.075 X-Spam-Level: * X-Spam-Status: No, score=1.075 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7, URIBL_BLACK=1.775] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4Avwm+aHgx2P for ; Sun, 6 Nov 2011 13:07:56 -0800 (PST) Received: from mail-fx0-f53.google.com (mail-fx0-f53.google.com [209.85.161.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 95505431FB6 for ; Sun, 6 Nov 2011 13:07:56 -0800 (PST) Received: by faan15 with SMTP id n15so1738525faa.26 for ; Sun, 06 Nov 2011 13:07:55 -0800 (PST) Received: by 10.223.17.3 with SMTP id q3mr41739585faa.28.1320613675216; Sun, 06 Nov 2011 13:07:55 -0800 (PST) Received: from localhost (dsl-hkibrasgw4-fe5cdc00-23.dhcp.inet.fi. [80.220.92.23]) by mx.google.com with ESMTPS id d22sm28327543fad.19.2011.11.06.13.07.52 (version=SSLv3 cipher=OTHER); Sun, 06 Nov 2011 13:07:53 -0800 (PST) From: Jani Nikula To: Austin Clements , notmuch@notmuchmail.org Subject: Re: [PATCH] Store "from" and "subject" headers in the database. In-Reply-To: <1320599856-24078-1-git-send-email-amdragon@mit.edu> References: <1320599856-24078-1-git-send-email-amdragon@mit.edu> User-Agent: Notmuch/0.9+32~gc0c32d1 (http://notmuchmail.org) Emacs/23.3.1 (i686-pc-linux-gnu) Date: Sun, 06 Nov 2011 23:07:51 +0200 Message-ID: <87k47dt0wo.fsf@nikula.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: notmuch@kismala.com X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Nov 2011 21:07:57 -0000 On Sun, 6 Nov 2011 12:17:36 -0500, Austin Clements wrote: > This is a rebase and cleanup of Istvan Marko's patch from > id:m3pqnj2j7a.fsf@zsu.kismala.com > > Search retrieves these headers for every message in the search > results. Previously, this required opening and parsing every message > file. Storing them directly in the database significantly reduces IO > and computation, speeding up search by between 50% and 10X. Hi, sounds good, but... > Taking full advantage of this requires a database rebuild, but it will > fall back to the old behavior for messages that do not have headers > stored in the database. ...what's the most convenient way of rebuilding the database while preserving my tags etc.? If this was merged, would an older version of notmuch choke on the rebuilt database with these headers? (To me it looks like it would be fine.) BR, Jani. > --- > lib/database.cc | 2 +- > lib/message.cc | 23 +++++++++++++++++++++-- > lib/notmuch-private.h | 11 +++++++---- > 3 files changed, 29 insertions(+), 7 deletions(-) > > diff --git a/lib/database.cc b/lib/database.cc > index fa632f8..e4ef14e 100644 > --- a/lib/database.cc > +++ b/lib/database.cc > @@ -1725,7 +1725,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch, > goto DONE; > > date = notmuch_message_file_get_header (message_file, "date"); > - _notmuch_message_set_date (message, date); > + _notmuch_message_set_header_values (message, date, from, subject); > > _notmuch_message_index_file (message, filename); > } else { > diff --git a/lib/message.cc b/lib/message.cc > index 8f22e02..ca7fbf2 100644 > --- a/lib/message.cc > +++ b/lib/message.cc > @@ -412,6 +412,21 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message) > const char * > notmuch_message_get_header (notmuch_message_t *message, const char *header) > { > + std::string value; > + > + /* Fetch header from the appropriate xapian value field if > + * available */ > + if (strcasecmp (header, "from") == 0) > + value = message->doc.get_value (NOTMUCH_VALUE_FROM); > + else if (strcasecmp (header, "subject") == 0) > + value = message->doc.get_value (NOTMUCH_VALUE_SUBJECT); > + else if (strcasecmp (header, "message-id") == 0) > + value = message->doc.get_value (NOTMUCH_VALUE_MESSAGE_ID); > + > + if (!value.empty()) > + return talloc_strdup (message, value.c_str ()); > + > + /* Otherwise fall back to parsing the file */ > _notmuch_message_ensure_message_file (message); > if (message->message_file == NULL) > return NULL; > @@ -795,8 +810,10 @@ notmuch_message_set_author (notmuch_message_t *message, > } > > void > -_notmuch_message_set_date (notmuch_message_t *message, > - const char *date) > +_notmuch_message_set_header_values (notmuch_message_t *message, > + const char *date, > + const char *from, > + const char *subject) > { > time_t time_value; > > @@ -809,6 +826,8 @@ _notmuch_message_set_date (notmuch_message_t *message, > > message->doc.add_value (NOTMUCH_VALUE_TIMESTAMP, > Xapian::sortable_serialise (time_value)); > + message->doc.add_value (NOTMUCH_VALUE_FROM, from); > + message->doc.add_value (NOTMUCH_VALUE_SUBJECT, subject); > } > > /* Synchronize changes made to message->doc out into the database. */ > diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h > index 0d3cc27..60a932f 100644 > --- a/lib/notmuch-private.h > +++ b/lib/notmuch-private.h > @@ -93,7 +93,9 @@ NOTMUCH_BEGIN_DECLS > > typedef enum { > NOTMUCH_VALUE_TIMESTAMP = 0, > - NOTMUCH_VALUE_MESSAGE_ID > + NOTMUCH_VALUE_MESSAGE_ID, > + NOTMUCH_VALUE_FROM, > + NOTMUCH_VALUE_SUBJECT > } notmuch_value_t; > > /* Xapian (with flint backend) complains if we provide a term longer > @@ -269,9 +271,10 @@ void > _notmuch_message_ensure_thread_id (notmuch_message_t *message); > > void > -_notmuch_message_set_date (notmuch_message_t *message, > - const char *date); > - > +_notmuch_message_set_header_values (notmuch_message_t *message, > + const char *date, > + const char *from, > + const char *subject); > void > _notmuch_message_sync (notmuch_message_t *message); > > -- > 1.7.2.3 > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch