From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Pip Cet Newsgroups: gmane.emacs.devel Subject: Re: Run (some) tests more automagically? Date: Sun, 21 Feb 2021 14:28:00 +0000 Message-ID: References: <87v9alk0xi.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="31443"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Lars Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Feb 21 15:30:14 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lDpkU-00087C-Q9 for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Feb 2021 15:30:14 +0100 Original-Received: from localhost ([::1]:49966 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lDpkT-0004px-RE for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Feb 2021 09:30:13 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:60424) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lDpj1-0004Kx-5b for emacs-devel@gnu.org; Sun, 21 Feb 2021 09:28:44 -0500 Original-Received: from mail-oo1-xc31.google.com ([2607:f8b0:4864:20::c31]:41275) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lDpiw-0008It-98 for emacs-devel@gnu.org; Sun, 21 Feb 2021 09:28:42 -0500 Original-Received: by mail-oo1-xc31.google.com with SMTP id h38so2381999ooi.8 for ; Sun, 21 Feb 2021 06:28:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=bN1kJqXZb54Oau2AAWCcVnYHPG6GZLP0w7MOVWpXYyY=; b=Xdk5IQutyyOV5GK3HVhJXqv6zTeRwDWIRVvEdEYUlatZEmk0m9fRUit5QHdGF04M48 189vQtI8bT3cZoqwoa/tQifLwr/FwNSSgKtX7/+U+IZq49KM0pVZMuF8btskIixCYOeW bNwc64FaVucziwU4mAsd1FBcisu0ZfvpH/NKd/n9hXQz9jeifTItHMZq6w4AbDyaTjNO 6He71xu4xN/SxNuVdMGqMeZkpyIaW2lsLhtD2P/4tKJUBBpktGoXSnvFLMz1eL/aVM9Q eqprBSqhdqisn/PhRYgLV6pwH9YBSKL8IH7L8+utL714/MPAaJj96518UpM8T211IH1r L8ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=bN1kJqXZb54Oau2AAWCcVnYHPG6GZLP0w7MOVWpXYyY=; b=XlqPW/hNLcXzzVwEQ0K9lOy6+zYREhyyNdDwcfvSq/bnNqsamXO9iOPYp3Kf6dAMda XRCQwmZ50C+3JcXSueAZ4VU7uZW7ygV2rpKMqr+wCV9rDqQxDOS8G9x/NRoT5/1mCu6M U5AHvf+t0rrMluBa6iHRUDpWyPK4Il0MvFjSl/xTAJoUp+OfCqCtW+tHfKgdq2/x5vky gVMD2VzVfYRIJmaeOGmzt5I1r0VDyYuDCx4sOIWSpKCeRVlZF9DXORDvzJJz4wjjdPA2 8xZm21zOYSvXvMSnbvj/07R7EAmXJD1Tm+dqKIIIHKV5jp+r/Eed++9REoH3G/JKgjLF PXHg== X-Gm-Message-State: AOAM532yijRJJKwmWuBbq1vvTETSImGno8lGdTVA6raVSYN52znMyQuD bS9mpoMh7/pL0uWvNpIWJxO4pJAlbR/08PW69GA= X-Google-Smtp-Source: ABdhPJzHq3YarQWxloy8FuoZp+H6XknK/iTUW5JNnqJ8uilA64jMLI/H33HZJlZ//ijB81fhFng2Qrg1NSvE4R7RohA= X-Received: by 2002:a4a:9873:: with SMTP id z48mr13583097ooi.44.1613917716219; Sun, 21 Feb 2021 06:28:36 -0800 (PST) In-Reply-To: <87v9alk0xi.fsf@gnus.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::c31; envelope-from=pipcet@gmail.com; helo=mail-oo1-xc31.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:265388 Archived-At: On Sun, Feb 21, 2021 at 1:27 PM Lars Ingebrigtsen wrote: > I wonder whether anybody's considered making "make" run some tests. > > It's a recurring issue that we make a change, and then we forget to run > the test suite. I mean, it's not a major problem, because it'll > eventually get run by somebody, but I'd feel more confident in my own > changes if I'd remember to run more tests more often. > > The reason we don't is because we don't want to wait for some minutes > while running the entire suite. That's certainly part of the reason for me. I find it important to remember that (to me) make is a tool to save time, and that requires compromise between remaking everything that could possibly have changed (which would require recompiling all elcs (and elns) for every change in the C sources, and also would require us to accurately model all potential ELisp-to-ELisp dependencies, some of which are circular) and just making things once. Doing the former is possible for projects that have flat dependency trees, not for deep (and possibly cyclic) dependency graphs. > So here's today's not-thought-out-at-all idea: Since tests for > lisp/foo.el live in test/lisp/foo-tests.el, could we add some Makefile > magic to automatically run foo-tests.el if lisp/foo.el has been changed? Yes, let's! Except we need to agree on what "has been changed" means. My initial idea would be to create a time stamp file the first time make is run in an emacs directory and consider only those source files which are newer than the time stamp, only if they're recompiled. But what should we do when we run "git pull"? Should the time stamp be updated for all tests after every make, for successful tests only, or only if all tests that were run were successful? Should the check run after the rest of make has (possibly after a "build successful, continuing automatically to run tests" message so the impatient can interrupt it at that point), at the same time as the rest of make (increasing time-to-test, IMHO) or asynchronously after make has finished? It's easy to get this wrong, and I half-expect I'll have to add "nocheck" to all my Emacs makefiles if the feature lands. > Running just those tests shouldn't take too much time, and would > probably cover 93% of the relevant code changes for foo.el. Speaking of code changes, one thing I'm worried about is that one developer on, say, Windows does something, forgets to adjust the tests so there's a spurious failure on GNU/Linux, and then the next GNU/Linux developer to touch the file will see a test failure that has nothing to do with their changes. I don't have a good solution for that one. > One disadvantage to doing this (if we find a way) is that "make > foo-tests" is often very noisy, so we'd need a way to run tests silently > and just report if anything actually fails unexpectedly, I guess. I think that's an advantage: tests should be silent by default, rather than producing all kinds of irrelevant "by the way, your headlight's working" noise. If we can help enforce that by running them in make, that would be great. Pip