From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: An Emacs benchmarking suite Date: Mon, 14 May 2018 12:30:37 -0400 Message-ID: References: <8336yypvq0.fsf@gnu.org> <83bmdmnskd.fsf@gnu.org> <0af8a67e-3403-8e81-a666-78b3d8da54b1@disroot.org> <838t8qnld0.fsf@gnu.org> <87a7t21na2.fsf@russet.org.uk> <83a7t2ky6v.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1526315371 10829 195.159.176.226 (14 May 2018 16:29:31 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 14 May 2018 16:29:31 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon May 14 18:29:27 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fIGLk-0002go-VT for ged-emacs-devel@m.gmane.org; Mon, 14 May 2018 18:29:25 +0200 Original-Received: from localhost ([::1]:46694 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGNs-0002Cq-2O for ged-emacs-devel@m.gmane.org; Mon, 14 May 2018 12:31:36 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:55910) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIGN7-0002Bz-Kz for emacs-devel@gnu.org; Mon, 14 May 2018 12:30:50 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIGN4-0008NO-Iu for emacs-devel@gnu.org; Mon, 14 May 2018 12:30:49 -0400 Original-Received: from [195.159.176.226] (port=39552 helo=blaine.gmane.org) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fIGN4-0008MS-Bd for emacs-devel@gnu.org; Mon, 14 May 2018 12:30:46 -0400 Original-Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1fIGKu-0001he-72 for emacs-devel@gnu.org; Mon, 14 May 2018 18:28:32 +0200 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 19 Original-X-Complaints-To: usenet@blaine.gmane.org Cancel-Lock: sha1:YhPRNwlSP5QtdUZYKHsq/9wu4CM= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 195.159.176.226 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:225291 Archived-At: >> So, something like time to create a big list (for CPU), and time to read >> a defined file (for IO). Then you could say "this file should parse in >> 1x IO base-line + 10x CPU base-line. > > You assume a linear scalability, but that is not necessarily so. > The ratio between performance indices of different codes could vary > depending on the build option and the underlying OS. Another issue is that performance measuring is notoriously difficult (even on an otherwise idle machine, let alone on some server that has other tasks running at the same time). So you might be able to catch the "10x slower" case easily with a fairly high confidence that the problem is indeed that the code got slower, but if you want to catch the "20% slowdown" with any kind of confidence (without being drowned in false positives), you'll need either a very tight control on the test runs, or a good statistical analysis. Stefan