From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oEQQKaanEWHSVAAAgWs5BA (envelope-from ) for ; Tue, 10 Aug 2021 00:09:42 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id CFR1JKanEWGEQgAA1q6Kng (envelope-from ) for ; Mon, 09 Aug 2021 22:09:42 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1D61DEE39 for ; Tue, 10 Aug 2021 00:09:42 +0200 (CEST) Received: from localhost ([::1]:42460 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mDDSn-0007Bk-3W for larch@yhetil.org; Mon, 09 Aug 2021 18:09:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60362) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mDDSd-0007Bb-SB for guix-devel@gnu.org; Mon, 09 Aug 2021 18:09:31 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:59897) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mDDSb-0000HU-SR; Mon, 09 Aug 2021 18:09:31 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0C34A27BC78; Mon, 9 Aug 2021 23:09:27 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id d1bb17aa; Mon, 9 Aug 2021 22:09:26 +0000 (UTC) References: <87bl6hbd05.fsf@cbaines.net> <87a6lwoqrs.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= , Ricardo Wurmus Subject: Re: Project direction with testing changes (branches and patches) In-reply-to: <87a6lwoqrs.fsf@gnu.org> Date: Mon, 09 Aug 2021 23:09:24 +0100 Message-ID: <87r1f2i82z.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, guix-maintainers@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628546982; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=p73C+4Sg8rbhIVOp+kNET8GLEtC17McRg/VxC7qWVog=; b=Ci9pxYqvhcMjxHRYeXdQGenoyfQeePpNrXWo8yAaKYzO8Sx0pTY9eCmnVSxgVVp8CP3GnQ vA1rifP6lUvMIBBCL/BKnuY+onXv5ezUUpUK1qieJsjWug7qJ44OXtp80hzPqZzPbvOuzv CZEtkyWFHcFgxeAaGslUZrW74dz6V8nGu/1WKM80d6V6AWivWnCDudwHI+ve81IarVehiX ilzVH22KaJxj8Cfh6tpEoy1AO1TlRrFVVsrJ8m+JpxQBV0dXLnqsaJw0/rgcAfvuzS+7QM NLktkbDvNzzAg+zPyV5FsSlzBsluNPZJ8K+/fiO/ZNbagNxSYYvXgKWyVUxxGA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628546982; a=rsa-sha256; cv=none; b=T9vNVt/LdS/+4DZwjf9e4337B0fyl2Jq46gsPXFRTcaY/UuJ9vlveUTNz5riKU0FZqLeU+ RfwowQ2Ye7JV2c0dF1QSFvI0G7gzVrYw65FhJPyWWUVxcVxTY0FjR9iw3HXF222mHQU/9F Z1udCvZ30eFgp42JZM1wg/F9NXOozBukTO0sMDFXiOBSpSTHIQ7iagcSqXDrlQqp7tar5b cdnVXehGqjAIFh/4C5A+eUgGL+mqma/FRU/Vx+mUNS0MnNE+T6sVa7AXrXIKLSOZjv+YiW zSOEDMeDyvmezB0oem8HXhxSIt9QwBySonsQQ4x2RH4m0+kExF4MQZ8IDpNlsA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.51 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 1D61DEE39 X-Spam-Score: -4.51 X-Migadu-Scanner: scn0.migadu.com X-TUID: NZltpKDoIeXQ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> So, I think I've recently switched to thinking about the problem as one >> of testing changes, rather than just testing patches. Since both patch >> series, and branches are used to propose changes, I think this makes >> sense. >> >> In abstract, when testing a change, I would break down the problem as >> follows: >> >> - You need to work out what's affected by the change, so that you can >> assess the impact >> >> - Once you know what's effected, you can then build those >> packages/system tests/... and compare the build statuses and outputs >> against some baseline >> >> - Then there's the general UI component, ideally a first time >> contributor would be able to take advantage of automatic feedback >> about a patch they submit. There's multiple other groups of users >> though, like patch reviewers, and committers for example. > > Makes sense to me. > > I agree that the first problem, seeing what=E2=80=99s affected by a chang= e, is > solved, but it=E2=80=99s still hard to get that info. I think we could h= ave a > special =E2=80=9Cskin=E2=80=9D for the Guix Data Service to make it easie= r for people to > view specifically this information. IMO the current UI has the upside > that it=E2=80=99s generic and exposes all the available information, but = it has > the downside that it=E2=80=99s generic and exposes all the available > information. :-) > > Or we could extend Julien=E2=80=99s Gitile=C2=B9 to include links from co= mmits to > lists of changed packages. > > The UI doesn=E2=80=99t have to be a web UI actually; we could use the Data > Service client interface at > > and write a new CLI, Emacs mode (similar to =E2=80=98M-x build-farm=E2=80= =99), or > something. > > =C2=B9 https://git.lepiller.eu/gitile Indeed, the technical side of detecting changes is solved, but I agree that there's work needed on making that information useful. >> I think the first two sub-problems are effectively solved. The Guix Data >> Service is able to determine the changes between two revisions (assuming >> it's processed them). The Guix Build Coordinator can then be used to >> build the relevant packages/system tests, and report that information >> back to the Guix Data Service. >> >> The UI part is much less certain, I've done some work with Patchwork, >> and I do have some ideas in mind, but there's still more thinking and >> work to do in this area. >> >> Before pressing on though, I think it would be good to know if this is a >> viable direction? > > I think we desperately need more automation, even more than when you > started working on this! > > I think a first step could be to make info from the Guix Data Service > more readily available, as suggested above. And from there we could > address #2 and #3. > > The Patchwork instance you maintain at > > does a large part of what we want, though the UI is not my favorite I > must say. ;-) I wonder if we could again make minimal changes to Mumi > so that it includes links to the relevant bits at the Data Service. > That=E2=80=99d make it more readily available. WDYT? I think using Patchwork or Mumi is viable, it would probably not be great to use both in the long term. The most useful thing for me would be to pick an approach. Patchwork is closer in terms of features I think, since it has an API for patch series and checks. I know mumi gained the ability to generate mbox files for patch series now though. I think the requirements in terms of Mumi would be to have some way of querying for patch series to test, or at least all/recent patch series. I suppose something could just ask for the 1st or latest series each time there's an email to a issue, that might be the simplest approach. Then there's the bits you describe about showing relevant information about whatever tests take place. I guess Mumi would need an API or some way to find out about that information, and then display it. Maybe that could happen in the form of emails with machine+human readable data that Mumi could interpret. Unfortunately I don't know much about the Mumi internals. Ricardo, are you able to comment on the feasibility and whether this direction would make sense? >> Currently, there's no automated testing of patches, and testing of >> branches is limited to the information that Cuirass provides on failed >> builds. What I'm proposing for the future is: using the Guix Data >> Service together with the Guix Build Coordinator to analyse the effects >> of changes, whether that be from a patch series or a branch. I realise >> that I've already been experimenting with this, what I'm mostly >> referring to here is moving towards this being the documented approach, >> maintained by the project, not just me. > > From an administrative standpoint, I very much agree that this sort of > infrastructure should be financially supported by the project rather > than by an individual, and documented so we can maintain it > collectively. Of course that involves a bit of overhead, and sometimes > we=E2=80=99re not all responsive when it comes to paperwork or sysadmin (= perhaps > these are not our preferred activities?), but still, it makes sense to > build that collectively. > > That=E2=80=99s my take. Thanks for the update! That sounds promising. I think trying to change up how branches (staging/core-updates) are tested is a good place to start. The concrete change I'm proposing is to use an instance of the Guix Data Service plus an instance of the Guix Build Coordinator to do the testing and builds, rather than Cuirass on ci.guix.gnu.org which is the current approach. The main advantages of that would be the comparison support from the Guix Data Service, and the build performance and reliability that the Guix Build Coordinator brings. The main disadvantage is probably the lack of an admin like interface similar to that of Cuirass (I think this can be remedied in the medium term though). This would then hopefully be a stepping stone to starting to test patches using the same infrastructure as well. It would be good to hear what people think, particularly the maintainers, then I can either get on trying to make it happen, or not, depending on the feedback. I don't particularly want to push ahead without a strong positive opinion. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmERp5RfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcHdA/+PwF+6GqYwyt1ImPWk+9dA4xy7CaYgFf9 Oz5OCQgbv3R5weu/JjIRcbHS4c0bU2WVH1ePFZbwcw0jUOaT3PhAICFFNm1K9jLB p+9DwqZx1ZK9n4Kcc7cpGQqm4n9PT4yxPCm56OMaYAbTQEgMMufZgJo8e5/969w3 Y5m5mONjtT8Y3RwhyG73lj7TODvxC3TkGJjexcdG3S8QumA/AIiEaFJIXPe/X6Oo BhTCLO0D8KDxD0HD175c3lxdt8YWJdJUqxUc++a4+2rMPH0tIHC07GCkkpRQlkpX w0mhxks7ildba3Nppc1d1tQSdu6fx7MolC0vW8r+UAkM/Ve/h6/ExRHDthvMbSYZ rNH/gc6XmycZlsm/heMVuOc8H8HcBDLyEuSJXPIj15R0Lb3DPxkYuhIV1lTal4TF wfTV+LHSSitFzAQ7Pl7flQNZb4CuY4FTfOGpbWEitBX+90KSsEGOLRV85VLY9Yu0 2JBvBI53Dzw2N9eHiXkoeuAEVHWCk/CX6ytJwZveTOZP3I/un4clBV7M+1ee1ddH T3B8omQCAGit986LMTVct5PFNt5fTNIN+x1nXa5MsaLShkC5hSsqZP6AD+gbwk6K e4DepTuyuyP+8yf6KRi9MfVav/9ZSux6zXrNBuWm2SKnPA84NaRqk/pr8KbpmLOJ m6UJH1QqaYI= =oHvZ -----END PGP SIGNATURE----- --=-=-=--