From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: phillip.lord@russet.org.uk (Phillip Lord) Newsgroups: gmane.emacs.devel Subject: Re: An Emacs benchmarking suite Date: Tue, 15 May 2018 14:24:58 +0100 Message-ID: <87fu2t6ogl.fsf@russet.org.uk> References: <8336yypvq0.fsf@gnu.org> <83bmdmnskd.fsf@gnu.org> <0af8a67e-3403-8e81-a666-78b3d8da54b1@disroot.org> <838t8qnld0.fsf@gnu.org> <87a7t21na2.fsf@russet.org.uk> <83a7t2ky6v.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1526390635 19763 195.159.176.226 (15 May 2018 13:23:55 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 15 May 2018 13:23:55 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) Cc: emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue May 15 15:23:51 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fIZvh-00050t-Mz for ged-emacs-devel@m.gmane.org; Tue, 15 May 2018 15:23:50 +0200 Original-Received: from localhost ([::1]:38018 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIZxo-0006GD-NQ for ged-emacs-devel@m.gmane.org; Tue, 15 May 2018 09:26:00 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:56298) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIZxc-0006Dq-6P for emacs-devel@gnu.org; Tue, 15 May 2018 09:25:49 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIZxW-0003H2-Es for emacs-devel@gnu.org; Tue, 15 May 2018 09:25:48 -0400 Original-Received: from cloud103.planethippo.com ([78.129.138.110]:47933) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fIZxW-0002eU-5c for emacs-devel@gnu.org; Tue, 15 May 2018 09:25:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=russet.org.uk; s=default; h=Content-Type:MIME-Version:Message-ID: In-Reply-To:Date:References:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=1W1hxSapE4P3KT71rkCq7pU+n0m0FLqqTsN1XRMj+Gk=; b=bzIPf3ilQBGMuvMM+b2ug/8ws kPz1lkmO7GW+3nmAdatSDM1E8aRLpVOHOK0yL/20fTA1KqX1V5eVcjds60poRRIEnsxKkEU9Bo7H2 HqYfZD8Y8T6GhNAJ+pKONtSP+QPKA45i2K5k71tnFAX1ESMVmf3u4DhrrwneKEFv/aQ+Qecuin8Gr +GxdK7TW4zWL/w6dicAKeCmwjBL5zdwYGGyb3VgafJjc95m33YPQcKwq6qnY6tjEcC+Jr6myuRX7a Ow1cqQ7xVkUDRcrwjE3GY62taNylTRSQRUB2GS6Uc7hrbK9oGqAqIFyarqrYV32ix7l8M8tXPla5z 7FlK+NsEg==; Original-Received: from janus-nat-128-240-225-124.ncl.ac.uk ([128.240.225.124]:47112 helo=russet.org.uk) by cloud103.planethippo.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1fIZwp-003Sh2-76; Tue, 15 May 2018 13:24:59 +0000 In-Reply-To: (Stefan Monnier's message of "Mon, 14 May 2018 12:30:37 -0400") X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cloud103.planethippo.com X-AntiAbuse: Original Domain - gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - russet.org.uk X-Get-Message-Sender-Via: cloud103.planethippo.com: authenticated_id: phillip.lord@russet.org.uk X-Authenticated-Sender: cloud103.planethippo.com: phillip.lord@russet.org.uk X-Source: X-Source-Args: X-Source-Dir: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 78.129.138.110 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:225307 Archived-At: Stefan Monnier writes: >>> So, something like time to create a big list (for CPU), and time to read >>> a defined file (for IO). Then you could say "this file should parse in >>> 1x IO base-line + 10x CPU base-line. >> >> You assume a linear scalability, but that is not necessarily so. >> The ratio between performance indices of different codes could vary >> depending on the build option and the underlying OS. > > Another issue is that performance measuring is notoriously difficult > (even on an otherwise idle machine, let alone on some server that has > other tasks running at the same time). So you might be able to catch > the "10x slower" case easily with a fairly high confidence that the > problem is indeed that the code got slower, but if you want to catch the > "20% slowdown" with any kind of confidence (without being drowned in > false positives), you'll need either a very tight control on the test > runs, or a good statistical analysis. Indeed, it would be very broad brush. Or, alternatively, it would be tied very tightly to a specific infrastructure. Phil