From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 20A92429E25 for ; Sun, 27 Nov 2011 10:41:03 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Al1HnpWyU5R for ; Sun, 27 Nov 2011 10:41:02 -0800 (PST) Received: from socrates.hocat.ca (socrates.hocat.ca [76.10.188.53]) by olra.theworths.org (Postfix) with ESMTP id 823DC431FB6 for ; Sun, 27 Nov 2011 10:41:02 -0800 (PST) Received: from hermes.hocat.ca (hermes.hocat.ca [69.165.170.253]) by socrates.hocat.ca (Postfix) with SMTP id 567CC14B1; Sun, 27 Nov 2011 11:40:58 -0700 (MST) Received: (nullmailer pid 9107 invoked by uid 1000); Sun, 27 Nov 2011 18:40:57 -0000 From: Tom Prince To: Notmuch Mail Subject: [PATCH] dump: Don't sort the output by message id. Date: Sun, 27 Nov 2011 13:40:53 -0500 Message-Id: <1322419253-9071-1-git-send-email-tom.prince@ualberta.net> X-Mailer: git-send-email 1.7.6.1 In-Reply-To: <1319884657-5574-1-git-send-email-thomas@schwinge.name> References: <1319884657-5574-1-git-send-email-thomas@schwinge.name> Cc: Thomas Schwinge X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Nov 2011 18:41:03 -0000 From: Thomas Schwinge Asking xapian to sort the messages for us causes suboptimal IO patterns. This would be useful, if we only wanted the first few results, but since we want everything anyway, this is pessimization. On 2011-10-29, a measurement on a 372981 messages instance showed that wall time can be reduced from 28 minutes (sorted by Message-ID) to 15 minutes (unsorted). Timings on 189605 messages: $ time notmuch.old dump 19.48user 5.83system 12:10.42elapsed 3%CPU (0avgtext+0avgdata 110656maxresident)k 3629584inputs+22720outputs (33major+7073minor)pagefaults 0swaps $ echo 3 > /proc/sys/vm/drop_caches $ time notmuch.new 14.89user 1.20system 3:23.58elapsed 7%CPU (0avgtext+0avgdata 46032maxresident)k 1256264inputs+22464outputs (43major+1990minor)pagefaults 0swaps --- This just moves the motivation to the commit message, and adds more detailed timing information. notmuch-dump.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/notmuch-dump.c b/notmuch-dump.c index 126593d..0475eb9 100644 --- a/notmuch-dump.c +++ b/notmuch-dump.c @@ -73,7 +73,10 @@ notmuch_dump_command (unused (void *ctx), int argc, char *argv[]) fprintf (stderr, "Out of memory\n"); return 1; } - notmuch_query_set_sort (query, NOTMUCH_SORT_MESSAGE_ID); + /* Don't ask xapian to sort by Message-ID. Xapian optimizes returning the + * first results quickly at the expense of total time. + */ + notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED); for (messages = notmuch_query_search_messages (query); notmuch_messages_valid (messages); -- 1.7.6.1