From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: phillip.lord@russet.org.uk (Phillip Lord) Newsgroups: gmane.emacs.devel Subject: Re: An Emacs benchmarking suite Date: Mon, 14 May 2018 12:37:09 +0100 Message-ID: <87a7t21na2.fsf@russet.org.uk> References: <8336yypvq0.fsf@gnu.org> <83bmdmnskd.fsf@gnu.org> <0af8a67e-3403-8e81-a666-78b3d8da54b1@disroot.org> <838t8qnld0.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1526297797 10954 195.159.176.226 (14 May 2018 11:36:37 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 14 May 2018 11:36:37 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) Cc: johnw@gnu.org, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon May 14 13:36:33 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fIBmJ-0002iD-SK for ged-emacs-devel@m.gmane.org; Mon, 14 May 2018 13:36:32 +0200 Original-Received: from localhost ([::1]:59621 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIBoP-0003yX-6p for ged-emacs-devel@m.gmane.org; Mon, 14 May 2018 07:38:41 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:58074) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fIBnq-0003yA-MV for emacs-devel@gnu.org; Mon, 14 May 2018 07:38:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fIBno-0008KX-0z for emacs-devel@gnu.org; Mon, 14 May 2018 07:38:06 -0400 Original-Received: from cloud103.planethippo.com ([78.129.138.110]:42535) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fIBnf-0007oQ-MU; Mon, 14 May 2018 07:37:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=russet.org.uk; s=default; h=Content-Type:MIME-Version:Message-ID: In-Reply-To:Date:References:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=LduFrbVHdf9TlfJQpljXdariBaBtyUl3AkaSNpJbuMQ=; b=28tv/Chc5m+DR37x1yfhiP6vV TSDkQQkUGuG9q0+KfC0mafIrh/uIwBdU+rXYEOHJkkgDAHTzHzIN/LPr6LxqYHGMpXe7hY09Rpdqr dmm4WiMtLt+SkJKJNDWtIR9S0EthI7pEslp7BnlaZjrLF5+P7VQyltxAtasQCvwyaTmR9OrAuZWbh kdKUZq2HUlgUA5r9oULjJ7f2tqBzWiDBnyuXjaaVOdkgUz7uSL3hUBYGzlXiJK93d7pPQEtqp0MOp kDPNTBKyn83RH0vmnKkrwcWZgerLIrQdfsIyBc+H0JKX/uxNiGqcCko2TWEBpMslmMsR8ngdzuI8T WYq5f0GBQ==; Original-Received: from janus-nat-128-240-225-124.ncl.ac.uk ([128.240.225.124]:46405 helo=russet.org.uk) by cloud103.planethippo.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1fIBmz-002ac2-Ap; Mon, 14 May 2018 11:37:13 +0000 In-Reply-To: <838t8qnld0.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 11 May 2018 20:38:35 +0300") X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cloud103.planethippo.com X-AntiAbuse: Original Domain - gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - russet.org.uk X-Get-Message-Sender-Via: cloud103.planethippo.com: authenticated_id: phillip.lord@russet.org.uk X-Authenticated-Sender: cloud103.planethippo.com: phillip.lord@russet.org.uk X-Source: X-Source-Args: X-Source-Dir: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 78.129.138.110 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:225285 Archived-At: Eli Zaretskii writes: >> Even so, how would one account for external factors in the operating >> system? Perhaps when the test is performed, the deltas between >> commits are given as percentages instead of CPU time. Once we have >> that, it would be great to have tests run multiple times and/or on >> various devices to refine the data further. > > Performance should indeed compare several versions n the same system. I wonder if there is a way to work out a base line value. Benchmarks could then look for multiples of this. So, something like time to create a big list (for CPU), and time to read a defined file (for IO). Then you could say "this file should parse in 1x IO base-line + 10x CPU base-line. That way the bench marks could become part of the test set. If things got significantly slower, tests would fail. Phil