From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lars Ingebrigtsen Newsgroups: gmane.emacs.devel Subject: Re: Run (some) tests more automagically? Date: Sun, 21 Feb 2021 17:05:04 +0100 Message-ID: <87pn0tiez3.fsf@gnus.org> References: <87v9alk0xi.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8252"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Pip Cet Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Feb 21 17:06:45 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lDrFs-00020s-Gw for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Feb 2021 17:06:44 +0100 Original-Received: from localhost ([::1]:45942 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lDrFr-0005DQ-JJ for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Feb 2021 11:06:43 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:49136) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lDrEX-0004TU-Df for emacs-devel@gnu.org; Sun, 21 Feb 2021 11:05:21 -0500 Original-Received: from quimby.gnus.org ([2a01:4f9:2b:f0f::2]:55080) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lDrEN-000857-EF for emacs-devel@gnu.org; Sun, 21 Feb 2021 11:05:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=jBYpn2ONgYx9zp1GMtkgbXSaZDTuRcK1Y/FdADa3RA4=; b=XlyuzNIBQSdezeLypYWkZkfW9O aJ5m77qqRjknqtLNzjkDdzGzbgoi4nrM3Z+M1i86/p/k5/b/iqZzOtIaDPGMXeYuCouHWXXasleQS TudVuzqk3fI/LEfuuhYD9CyNXkmhI6JIsW9JdHyPwdvsvyZT6vbXYnYayq6TfzZMGzZ8=; Original-Received: from cm-84.212.220.105.getinternet.no ([84.212.220.105] helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lDrEI-0001qp-1A; Sun, 21 Feb 2021 17:05:08 +0100 Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAACBj SFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAElBMVEUhHB0oIyNYSEKT dmzjwrX////3c5ssAAAAAWJLR0QF+G/pxwAAAAd0SU1FB+UCFQ8sElaOiYcAAAFFSURBVDjL7ZPh bcQwCIXNTQB0gfDcAXomG8T7z1Sw3bu0jdRK/VtOFyl8BgyPlHJlwuXaSMofTAFI2DdQ3f0SqAzj XyT/tx+ML710OfdYgXTnP8LOnEiWJlyIP503WDxSrdOarNOCIOf9mWmYxA3plJM/VDeOCLV0LL/O 3RmXUqOP4jRyR4DyAEyrUeJ05i+upYZIJLMBEpjFKlbPEmChBVQRx3MZ3SoQd5gg8uDFsPfevakA PCOyYByrPS2BzlneMi2g+wCe5SdQgjzB0eS2gJDlxzEzRa54m7PPHgBf/n4sMMYaAfsJTCGjmczU H2a2FA5Qqz/Bm+W1hh9fQCqQSg/wqNE3HaCMtmtbId59yxnLECmLj0aOBm+sdcgrCWq8x6c+RN1C A1iqB7Qo0kAcJ2L+CVA4tikmu+/Nt839db/fQ2boO+f/RiO2cVKPAAAAJXRFWHRkYXRlOmNyZWF0 ZQAyMDIxLTAyLTIxVDE1OjQ0OjE3KzAwOjAwJUu6GAAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMS0w Mi0yMVQxNTo0NDoxNyswMDowMFQWAqQAAAAASUVORK5CYII= X-Now-Playing: Console's _Herself_: "Walking The Equator" In-Reply-To: (Pip Cet's message of "Sun, 21 Feb 2021 14:28:00 +0000") Received-SPF: pass client-ip=2a01:4f9:2b:f0f::2; envelope-from=larsi@gnus.org; helo=quimby.gnus.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:265401 Archived-At: Pip Cet writes: > Yes, let's! Except we need to agree on what "has been changed" means. > My initial idea would be to create a time stamp file the first time > make is run in an emacs directory and consider only those source files > which are newer than the time stamp, only if they're recompiled. Hm, yes, a time stamp would probably be the way to go... > But what should we do when we run "git pull"? Should the time stamp be > updated for all tests after every make, for successful tests only, or > only if all tests that were run were successful? Should the check run > after the rest of make has (possibly after a "build successful, > continuing automatically to run tests" message so the impatient can > interrupt it at that point), at the same time as the rest of make > (increasing time-to-test, IMHO) or asynchronously after make has > finished? These are all good questions. :-) Off the top of my head without thinking about it more than ten seconds: Since this is a low-cost low-effort thing, I think the timestamp should just be updated when it's run, and just a single timestamp, no matter how much fails. So if you do a git pull, and that updates foo.el and bar.el, then "make" would (as now) compile foo.el and bar.el, and Emacs would then notice that foo.elc and bar.elc are newer than the previous timestamp, and then run the foo-tests.el and bar-tests.elc files, and then update the timestamp. I think that might be unnoticeable enough that developers wouldn't be disabling this when working? Of course, if we rely on this too much, then there's the temptation to never actually run a full "make check", but we have the CI for that, so perhaps that's not too big a worry. > Speaking of code changes, one thing I'm worried about is that one > developer on, say, Windows does something, forgets to adjust the tests > so there's a spurious failure on GNU/Linux, and then the next > GNU/Linux developer to touch the file will see a test failure that has > nothing to do with their changes. I don't have a good solution for > that one. Me neither. But that's also the case today. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no