From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Gemini Lasswell Newsgroups: gmane.emacs.devel Subject: a benchmark suite for Emacs Date: Sun, 25 Nov 2018 17:35:12 -0800 Message-ID: <87efb8iptb.fsf@runbox.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1543196056 24297 195.159.176.226 (26 Nov 2018 01:34:16 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 26 Nov 2018 01:34:16 +0000 (UTC) User-Agent: mu4e 1.0; emacs 26.1.90 To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Nov 26 02:34:12 2018 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gR5mt-00069m-KE for ged-emacs-devel@m.gmane.org; Mon, 26 Nov 2018 02:34:11 +0100 Original-Received: from localhost ([::1]:33769 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gR5oz-0005Vz-Th for ged-emacs-devel@m.gmane.org; Sun, 25 Nov 2018 20:36:21 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:49256) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gR5oI-0005Vg-EV for emacs-devel@gnu.org; Sun, 25 Nov 2018 20:35:39 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gR5oF-0002Mi-Br for emacs-devel@gnu.org; Sun, 25 Nov 2018 20:35:38 -0500 Original-Received: from aibo.runbox.com ([91.220.196.211]:55566) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gR5oF-0002M1-1c for emacs-devel@gnu.org; Sun, 25 Nov 2018 20:35:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=rbselector1; h=Content-Type:MIME-Version:Message-ID:Date:Subject:To:From; bh=QMYdF3zPM/4gPhHxlCY1Q2YbCMcxjLXYMqC3ohOH87o=; b=VP0SMsvmsCAjzIoUShYH4rLL6 RiYh9005tR9xUcdY5fHmzU+3vfjnL4UHOqIOraWUOhnqjlHQYZ6DQWS58LOCYI1ygxXoB8xavV1oO yd+V/SaSc+yLsFH+k797zXeGtIs6NZBMNSfSaOCjh7UrhchgteGA13VdQgGCo6/K4kj4N7fl9DaQe gC7hWSN4+U5xratOdKNZUlJScrmGZrC337b79YqJPXcvTChXbliCj1xCmdxNoGlVO7MMlp+QHllhj 8LlL1pZf6+OBv4u/jpj07zNokDHlYcFACxq5lLQpfbaLQomKbEbuhTasplZosAacWF0KKUm3bIFuk B83KjVn2A==; Original-Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by mailtransmit02.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1gR5oC-0007r6-Tc for emacs-devel@gnu.org; Mon, 26 Nov 2018 02:35:33 +0100 Original-Received: by mailfront11.runbox.com with esmtpsa (uid:179284 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1gR5nu-0002MH-W5 for emacs-devel@gnu.org; Mon, 26 Nov 2018 02:35:15 +0100 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 91.220.196.211 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:231376 Archived-At: Hello all, I've just pushed to scratch/benchmarks my WIP benchmarking suite for Emacs and Emacs packages. It's a long ways from done. Some major missing parts are: - Better/more complete user interface - Documentation - Statistical analysis (report on noisiness of data) - Tramp integration/run benchmarks remotely - Use VC rather than calling git directly - More benchmark tasks - Tests But, given: - a local clone of a git repo - a program/shell script that produces an executable given that git repo and a commit - a directory in the git repo with Lisp files containing tasks to be benchmarked - a range of commits (as for git-rev-list(1)) - a number of commits to select from the range it can use the shell script to build older versions of Emacs from commits selected from the commit range given, run the tasks in those older Emacs, record timing data, errors, process output and messages, and then produce a report in the form of an org-mode file ready for org-babel-gnuplot. The benchmark data is stored in a directory hierarchy organized so that benchmark data created on separate machines can be easily git-merged together. There's no documentation yet, but if any of you have time and interest in running benchmarks, let me know and I'll write up instructions. For those of you who generously spend your time reviewing code, the new benchmark directory, thread.el and erb-task.el are ready for your attention, but erb.el is not. I also welcome ideas for more benchmarks, or anyone who wants to write some.