From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Mattias_Engdeg=C3=A5rd?= Newsgroups: gmane.emacs.devel Subject: Re: elisp-benchmarks Date: Thu, 10 Feb 2022 13:12:18 +0100 Message-ID: <12E6EB56-966F-49F9-B6FC-13C2668F041B@acm.org> References: Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6664"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Emacs-Devel To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Feb 10 13:16:26 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nI8N8-0001Uq-GA for ged-emacs-devel@m.gmane-mx.org; Thu, 10 Feb 2022 13:16:26 +0100 Original-Received: from localhost ([::1]:33402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nI8N7-0005Bs-4v for ged-emacs-devel@m.gmane-mx.org; Thu, 10 Feb 2022 07:16:25 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:53152) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nI8JN-0002Hw-QO for emacs-devel@gnu.org; Thu, 10 Feb 2022 07:12:33 -0500 Original-Received: from mail1468c50.megamailservers.eu ([91.136.14.68]:38630 helo=mail268c50.megamailservers.eu) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nI8JH-0000Vm-PA for emacs-devel@gnu.org; Thu, 10 Feb 2022 07:12:33 -0500 X-Authenticated-User: mattiase@bredband.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=megamailservers.eu; s=maildub; t=1644495141; bh=4J/J+dxouhr1d+DH26VBtd8NEl47B/B47frAdShERk4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From; b=Gz5+bhQ3FwGb3poRXA2iQ21TSVKmEZX3gfUA8QWb8jglXWs/N89FpJxCi06C2X47Y JqOec+sG9E5ImZOsBtbiwRv+b32JRxXcbFVgV9JOFtM65BMLh86e7Gj13YaT3s/iCB hCdeIDMTAIAqgp9mJuZaMF8FGzQbZieFzwySB61I= Feedback-ID: mattiase@acm.or Original-Received: from smtpclient.apple (c188-150-171-71.bredband.tele2.se [188.150.171.71]) (authenticated bits=0) by mail268c50.megamailservers.eu (8.14.9/8.13.1) with ESMTP id 21ACCJw0026440; Thu, 10 Feb 2022 12:12:20 +0000 In-Reply-To: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-CTCH-RefID: str=0001.0A742F15.62050125.0030, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CTCH-VOD: Unknown X-CTCH-Spam: Unknown X-CTCH-Score: 0.000 X-CTCH-Flags: 0 X-CTCH-ScoreCust: 0.000 X-Origin-Country: SE Received-SPF: softfail client-ip=91.136.14.68; envelope-from=mattiase@acm.org; helo=mail268c50.megamailservers.eu X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:286132 Archived-At: 9 feb. 2022 kl. 23.19 skrev Stefan Monnier : > And we see that Matthias's recent improvements to the bytecode > interpreter do make a quite significant difference on several of those > microbenchmarks ;-), and also on the bytecompiler benchmark = (offsetting > the extra work needed for the symbol-with-positions) tho they don't = make > much of a difference when it comes to scrolling(with-jit-lock) or when > it comes to reindenting code with SMIE :-) You are much too kind; my own macro-benchmarks indicate that we haven't = quite reached status quo ante yet. However I would like to caution against using the 'total' line of these = benchmarks and in fact suggest that it be removed entirely since it can = be very misleading: it is in effect tantamount to a completely arbitrary = weighting of the individual benchmarks. If we want an aggregate number that weighs all benchmark components = equally, one way is to first establish a baseline, use relative changes = to that baseline, and take the geometric average of those. But that only makes sense if we value each component equally and there = is no reason to do that -- many of the benchmarks measure essentially = the same thing, and the mixture cannot in any way be defended as = representative of any kind of practical Emacs use. Individual benchmarks can of course be of interest: a proper = presentation would be in a table where they each can be compared across = different changes. Rearranging your date and normalising to before = sympos, separately for timings that exclude and include GC, gives: | | ex gc | inc gc | | test | sympos | master | sympos | master | |--------------------+--------+--------+--------+--------| | bubble | 0.95 | 0.74 | 1.00 | 0.91 | | bubble-no-cons | 1.01 | 0.83 | 1.01 | 0.83 | | bytecomp | 1.06 | 0.99 | 1.06 | 1.02 | | dhrystone | 1.03 | 0.80 | 1.03 | 0.80 | | eieio | 0.99 | 0.83 | 0.99 | 0.90 | | fibn | 0.99 | 0.69 | 0.99 | 0.69 | | fibn-named-let | 0.98 | 0.68 | 0.98 | 0.68 | | fibn-rec | 1.01 | 0.66 | 1.01 | 0.66 | | fibn-tc | 1.02 | 0.60 | 1.02 | 0.60 | | flet | 1.06 | 0.68 | 1.06 | 0.68 | | inclist | 1.07 | 0.98 | 1.07 | 0.98 | | inclist-type-hints | 1.07 | 0.98 | 1.07 | 0.98 | | listlen-tc | 1.00 | 0.73 | 1.00 | 0.73 | | map-closure | 1.02 | 0.69 | 1.02 | 0.69 | | nbody | 0.97 | 0.85 | 0.99 | 0.97 | | pack-unpack | 0.99 | 0.80 | 0.99 | 0.90 | | pack-unpack-old | 1.01 | 0.91 | 1.01 | 0.94 | | pcase | 0.87 | 0.86 | 0.87 | 0.86 | | pidigits | 0.93 | 0.86 | 0.96 | 0.92 | | scroll | 0.97 | 1.04 | 0.97 | 1.04 | | smie | 1.02 | 1.01 | 1.01 | 1.01 | which is a bit more informative (at least if you know what the = benchmarks do, but otherwise it's all nonsense numbers anyway). For the record, my own Relint benchmark (we all have our pets!) is at = about 1.03 from the same baseline which is notable because it exercises = a wide variety of operations, many of which weren't affected by any of = the changes. > The >10% slowdown recently seen on the test suite is still a mystery > waiting for someone to figure out what's going on. A config option to switch off sympos would be handy. > BTW, I think one thing is clear when I look at those benchmarks: > Emacs's GC is not good enough. Indeed it's the elephant in the room. In addition, designing meaningful = benchmarks that aren't GC-dominated can be tricky.