From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ONdiC32KBmH7ZQEAgWs5BA (envelope-from ) for ; Sun, 01 Aug 2021 13:50:21 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 4M36Bn2KBmF/QQAAbx9fmQ (envelope-from ) for ; Sun, 01 Aug 2021 11:50:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CA48E128FB for ; Sun, 1 Aug 2021 13:50:20 +0200 (CEST) Received: from localhost ([::1]:33396 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mA9z1-0006hX-To for larch@yhetil.org; Sun, 01 Aug 2021 07:50:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50122) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mA9ys-0006hP-PR for guix-devel@gnu.org; Sun, 01 Aug 2021 07:50:10 -0400 Received: from mira.cbaines.net ([212.71.252.8]:58064) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mA9yq-0004O2-Hr; Sun, 01 Aug 2021 07:50:10 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id C56AF27BC6B; Sun, 1 Aug 2021 12:50:05 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 561f6480; Sun, 1 Aug 2021 11:50:05 +0000 (UTC) User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org, guix-maintainers@gnu.org Subject: Project direction with testing changes (branches and patches) Date: Sun, 01 Aug 2021 12:50:02 +0100 Message-ID: <87bl6hbd05.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1627818620; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=bcyXoIZTbSkFIKi0KBEiclAxdzoY91Y+fr2kre/M3HA=; b=t2efQS7i06hMK6ou7jyqlBiMx63iXLkHo4rpDm/7JcYYqBw0Jj2PAOIYdJ2Fe4S5slRiFY N++d/Qzvysvj2xKaWz7rgDdevBuw/bYGms7DqJx/bezGTlvKxztVbpqYCBe/dxgJKuxLwu n9Xtx0dQ26Wm94pwve2rlCeKpUXjYBZc56nXYBzWGi89uogIKJ0Gn0Fu4hi1IMmGEqUJ8d jr3YmqAbgfcgXMeGt4n0GQJjID/RMmG+JZFC86TScvF347LsV4SyCUhaW+LibiJipfrT4g 6zVTN08Iu/6+SNT9mgzppib7+Fv7wZY9wfNEKR1mZcofghmFMiNzcLXXL1wtwQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1627818620; a=rsa-sha256; cv=none; b=a9wTGGT7n05nujMOsrsaDTw4tN82tzWdms0uw3XntlcLd9QXzzzkLV3qmWZq67bdoXTdCK enjkWAuHRW7l5aZke1iJbyC9xG8XGN4hO/vkJ4dqcqJo3F0wSvDm9VQHxZ++uEQFdQP/20 RNmxBG2wSxcuKe1bpRisfB5bsj3F3ieED+/Q8lnaWLsgIYhZ5WOkLbPQfHH1Pqv9bombQC p3FRZBJcHjvsDuYrCuS9Bs4W+DIeUmfV9ntMtsU4NxNlTorD41EkNTA5roDS5vjcN31zhy eonXXF/yB0EpVcevG8VNcvIyEokwbBdh9xUUPF5kb3yQNUrigLlBUpaW3OCNjQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: CA48E128FB X-Spam-Score: -4.52 X-Migadu-Scanner: scn1.migadu.com X-TUID: lABW2fdGbcgK --=-=-= Content-Type: text/plain Hey, This is sort of a followup to [1], at least I think that's the last main email I sent out about testing changes (although I didn't use that term). I did also send out some notes from the Guix Day event back in February 2021 though [2]. 1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00583.html 2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00125.html Back in early 2020, I managed to start work on the Guix Build Coordinator [3]. That was meant to enable running reliable and performant substitute servers, but also meant to enable the kind of testing and quality assurance work that I had been thinking about, mostly through the perspective of testing patches. 3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html Getting the benefits to users didn't go as smoothly as I'd hoped, but since bordeaux.guix.gnu.org [4] launched back in June, there's a chance that the work on the Guix Build Coordinator has benefited users of Guix through improved substitutes. 4: https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/ As I said in [1], I did do some work last year to use the Guix Build Coordinator for testing patches and branches. Unfortunately the setup I'm using is currently not operating, I was having issues with running out of disk space on the main server, and I haven't got around to spending the time/money to resolve that. I want to get another iteration of the patch testing setup working, but recent experiences with working on providing substitutes has made me think that discussing the direction with maintainers and as a project is almost more important. So, I think I've recently switched to thinking about the problem as one of testing changes, rather than just testing patches. Since both patch series, and branches are used to propose changes, I think this makes sense. In abstract, when testing a change, I would break down the problem as follows: - You need to work out what's affected by the change, so that you can assess the impact - Once you know what's effected, you can then build those packages/system tests/... and compare the build statuses and outputs against some baseline - Then there's the general UI component, ideally a first time contributor would be able to take advantage of automatic feedback about a patch they submit. There's multiple other groups of users though, like patch reviewers, and committers for example. I think the first two sub-problems are effectively solved. The Guix Data Service is able to determine the changes between two revisions (assuming it's processed them). The Guix Build Coordinator can then be used to build the relevant packages/system tests, and report that information back to the Guix Data Service. The UI part is much less certain, I've done some work with Patchwork, and I do have some ideas in mind, but there's still more thinking and work to do in this area. Before pressing on though, I think it would be good to know if this is a viable direction? Currently, there's no automated testing of patches, and testing of branches is limited to the information that Cuirass provides on failed builds. What I'm proposing for the future is: using the Guix Data Service together with the Guix Build Coordinator to analyse the effects of changes, whether that be from a patch series or a branch. I realise that I've already been experimenting with this, what I'm mostly referring to here is moving towards this being the documented approach, maintained by the project, not just me. So yes, is this something that people want, or don't want? If you're uncertain and have questions, it would be good to know what those questions are? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmEGimpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcToA//dRZQv1G+YVLbq449HBpKyjCtn139TDnJ neQo0wAJrHVE8MbEjxBG3khqSASLLmpPtK3zCqSkEMfM3P++8Ai+mMiIKwGkU95O d8BqNkDLy5noWMSrjCXd5mA9ekaL/IL0/Anx1C21Af2bZU3RT7DO3hgeLss+/KdX b3cL67ZzT6heO+aYKBFJEv9FiQZOoUkInb64bHhUaK6kqpcsnhKGWEEz3HscraOO jZt95GRi4ACtQS/X9CJHm9QGvOX99YQUGFikmI5V/gTRkfVkrY4SIAGcWkdCJaAJ hwwPSWZePqntVW1W2qtWoZ+GutspX0D5x5Ou/vrCnjyGse9/GLYYIVIS9DZP5gpA aUSWZgGIEcHOK1cTiSs+UkfEclZpkLAj5LNgCIB70Do5j9YGEPexwBX/ecsdF1bp aKWAWDaD006TXLgKcapwjTVZSS7eYdmJU1HWCHqpdyjkvjJkUqW09PaVShBeYkG0 8YOEQWl7P+NKHxZIPvy44CwCsJi1N4A0JYABEehYgKqiipPkzvLANo2L/3435Yxy aDlfSRKQRoqMYAw7UcUuHnwYpgOcfbWOMKvf7qpJ82B1XBsR027ZMB8y1ARgAQbu aCXpvGU8MDmOZ0/Dj15Z9VvakZZsbWD7FJFoFxFJNFVAiktCtFppmWn5zdTd38zz V9Dd7aloVoQ= =RkIC -----END PGP SIGNATURE----- --=-=-=--