From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 08D53431FAF for ; Sun, 25 Nov 2012 16:05:53 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DG3h5vQDF2tl for ; Sun, 25 Nov 2012 16:05:52 -0800 (PST) Received: from tesseract.cs.unb.ca (tesseract.cs.unb.ca [131.202.240.238]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 78C52431FAE for ; Sun, 25 Nov 2012 16:05:52 -0800 (PST) Received: from fctnnbsc30w-156034089108.dhcp-dynamic.fibreop.nb.bellaliant.net ([156.34.89.108] helo=zancas.localnet) by tesseract.cs.unb.ca with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1TcmCs-0007pZ-Vu; Sun, 25 Nov 2012 20:05:51 -0400 Received: from bremner by zancas.localnet with local (Exim 4.80) (envelope-from ) id 1TcmCm-0005SZ-JH; Sun, 25 Nov 2012 20:05:44 -0400 From: David Bremner To: Austin Clements Subject: Re: [Patch v4 2/2] test: initial performance testing infrastructure In-Reply-To: <20121125214023.GO4562@mit.edu> References: <1353855745-11697-1-git-send-email-david@tethera.net> <1353855745-11697-3-git-send-email-david@tethera.net> <20121125214023.GO4562@mit.edu> User-Agent: Notmuch/0.14+75~g984212d (http://notmuchmail.org) Emacs/24.1.1 (x86_64-pc-linux-gnu) Date: Sun, 25 Nov 2012 20:05:42 -0400 Message-ID: <87obilut2h.fsf@zancas.localnet> MIME-Version: 1.0 Content-Type: text/plain X-Spam_bar: - Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 00:05:53 -0000 Austin Clements writes: >> +subdirs := compat completion emacs lib man parse-time-string >> +subdirs := $(subdirs) performance-test util test > += ? > Sure. >> +CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz > > Would it make sense to split out the different size corpora so a user > could, say, only download the small one? Currently the choice of test is local to given test file; one doing something particularly intense (or just lots of repetitions) might want to only use a subset. So I'm not sure if separate downloading of smaller corpora makes sense. This is all hypothetical at the moment, since the one test file uses the full corpus. > "\nPlease download ${TXZFILE} using\n\n"? OK >> +add_email_corpus takes arguments "--small" and "--medium" for when you >> +want smaller corpuses to check. > > "corpora"? reworded to say ,---- | add_email_corpus takes arguments "--small" and "--medium" for when you | want smaller subsets of the corpus to check. `---- > > I'm a bit confused by this. What happens if you don't specify --small > or --medium? Is the "large"/default corpus just the combined small > and medium corpora? Would be worth a comment, at least. Hopefully the README makes this clear(er) now? > This probably doesn't matter now, but I wonder if we want to unpack on > first use to somewhere not test-specific and then cp -rl the corpus > into the test directory. I haven't tried unpacking the corpus yet, > but if you're running tests repeatedly to compare results, or running > more than one performance test, it seems like a full decompress and > unpack could get onerous. Hmm. On my machine it is 10s for the copy versus 45s for a full unpack. For some reason I tested with "cp -a" which is incredibly slow, so I thought there was no loss. For comparison the basic test takes about 10 minutes on the same machine. In any case this can wait until we have a second test file and a second call to add_mail_corpus, adding caching now would not help.