From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Kangas Newsgroups: gmane.emacs.devel Subject: Re: New "make benchmark" target Date: Mon, 30 Dec 2024 12:58:23 -0600 Message-ID: References: <87h679kftn.fsf@protonmail.com> <87y107g0xc.fsf@protonmail.com> <87frm51jkr.fsf@protonmail.com> <861pxpp88q.fsf@gnu.org> <87frm5z06l.fsf@protonmail.com> <86msgdnqmv.fsf@gnu.org> <87wmfhxjce.fsf@protonmail.com> <86jzbhnmzg.fsf@gnu.org> <87o70txew4.fsf@protonmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="18137"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , mattiase@acm.org, eggert@cs.ucla.edu, emacs-devel@gnu.org To: Andrea Corallo , Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Dec 30 19:59:13 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tSKyv-0004d2-4O for ged-emacs-devel@m.gmane-mx.org; Mon, 30 Dec 2024 19:59:13 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tSKyE-0002h5-OV; Mon, 30 Dec 2024 13:58:30 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tSKyD-0002gt-1f for emacs-devel@gnu.org; Mon, 30 Dec 2024 13:58:29 -0500 Original-Received: from mail-ed1-x533.google.com ([2a00:1450:4864:20::533]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tSKyB-0007Fz-Ik; Mon, 30 Dec 2024 13:58:28 -0500 Original-Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-5d3e6f6cf69so15740468a12.1; Mon, 30 Dec 2024 10:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735585105; x=1736189905; darn=gnu.org; h=cc:to:subject:message-id:date:mime-version:references:in-reply-to :from:from:to:cc:subject:date:message-id:reply-to; bh=qYBWyxxlOBzHYIBoW38YhDxvDNPFHUPCGuIXTJZczdI=; b=fXvz6RaG6Mt0OsKF2lWbcATIhvpzotfGoWSOSFv2GwesKHw5kTC1JrQgqgwomLhyFI thOCVPFKampOTjvYAQVUkoM1f4XpqBaFLbHrmeuTMrgXiXS13forikEg/XyTLeVUCtcF Qnzh8xd1tEP8pUlV/7d8xPpC37iPRDJVf32DOHFDnuFJ7vzKNU7Yf/gq8OE5hN7c/i8l hkzClLlL7yr2Sk5oFaGuKsIzrfFeXw33IzuB1rkcFrVSSofRN+ccGr9jkJt6Vy+GF+Oh CN3LPx8uGaCU/dOXU2M1Qacbo/MuE6EHud6lFXhLzjePmPFqGNv65IMk6YQcypAEbnG8 cNpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735585105; x=1736189905; h=cc:to:subject:message-id:date:mime-version:references:in-reply-to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qYBWyxxlOBzHYIBoW38YhDxvDNPFHUPCGuIXTJZczdI=; b=XdxEqJQShz2ymo6jO8MSyhJRGJBMdThBsJJ2Ee3qPt8IhOyVriE3eagBAclBJ6UpGD 11kUz1CEqlLru5KHAGqcOBKOcMXC+7tKJxh/clxdo6M7vX9/tFzZvNdIRycKnDA+HwTX lEKgJuyDzxOnvkUjagHkTG264z7HIHHBWX3NejlbqUe3P2XpiNfW3d3hIyEgWI6OGzXs o3WOp7+AxxZ1lMQN4/32SsmriJz9njtz2DUvUjkYwLOAEivvDWxUxzCN4XBNTE2Fh3xj wIpUa7TXm+LGeB7IxesnBvmmcjdzktWvttgJdpNFdsjDSH3Y3NOlkjr2pbchBK/DV8+1 yGTg== X-Forwarded-Encrypted: i=1; AJvYcCU4WDOjq6WuFUW2MnKDL+fVZ9ESrf5T0FqKFqYQEHh9MNZ7QOnovtjB6qmdSx3om56Q0ILLoS7hxJFUHA==@gnu.org X-Gm-Message-State: AOJu0YyOFnczBw/qM+1pb1KGuQUBPq5qIOVvSdhINNRI5Ofc//CjV+t8 N4vI3698jEfFPzacaa5iduzhVYZ4NCNTCwQqt/LuiUFFadBSdI/wqz9/wEJ9ykJcLCPPwI8jPeT EzmZBUuBO20BKpEVNoDEFTqxF4pRj3VEK X-Gm-Gg: ASbGncu8e5gNl49MZkOM+J6Cra59z1I2JBDRrfHyx9nDknjT9OuLBTCUCKlU03v5FfP Iv9c1Ak4bXxG9fwnYS4eiqwRLp8TULIPdvmOPA1Uo X-Google-Smtp-Source: AGHT+IEVNJIR6/DGwXLYNzu4vy3FHa6GXtCY/KGYApBIeLLiidd3+2/Rf0dMZ9XsSmEYsnmX1jldGhMJEHMH2bFE8iI= X-Received: by 2002:a17:907:6d16:b0:aab:97db:4b45 with SMTP id a640c23a62f3a-aac3444e8a7mr3234172466b.37.1735585104542; Mon, 30 Dec 2024 10:58:24 -0800 (PST) Original-Received: from 753933720722 named unknown by gmailapi.google.com with HTTPREST; Mon, 30 Dec 2024 12:58:23 -0600 In-Reply-To: Received-SPF: pass client-ip=2a00:1450:4864:20::533; envelope-from=stefankangas@gmail.com; helo=mail-ed1-x533.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:327464 Archived-At: Andrea Corallo writes: > Yes, elisp-benchmark is running all the selected benchmarks at each > iteration, so that a single one cannot take advantaged of the initial > cool CPU state. If unstable throttling on a specific system is a > problem this will show up as computed error for that test. If a system > is throttling the right (and only) thing to do is to measure it, this is > in my experience what benchmarks do. > > That said tipically Eli is right, the typical use of a benchmark suite > is to run it as a whole and look at the total results, this indeed > accounts for avg throttling as well. May I propose writing down some general usage notes along the lines of the above somewhere?