From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Marusich Subject: Re: Treating tests as special case Date: Fri, 06 Apr 2018 01:58:35 -0700 Message-ID: <87lge0aeys.fsf@gmail.com> References: <20180405052439.GA30291@thebird.nl> <20180405082115.60e604a6@alma-ubu> <20180405084300.GB31585@thebird.nl> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42435) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f4NCu-0006f1-DT for guix-devel@gnu.org; Fri, 06 Apr 2018 04:58:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f4NCr-0006y7-9p for guix-devel@gnu.org; Fri, 06 Apr 2018 04:58:52 -0400 Received: from mail-it0-x235.google.com ([2607:f8b0:4001:c0b::235]:55024) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f4NCr-0006xp-2O for guix-devel@gnu.org; Fri, 06 Apr 2018 04:58:49 -0400 Received: by mail-it0-x235.google.com with SMTP id h143-v6so936431ita.4 for ; Fri, 06 Apr 2018 01:58:48 -0700 (PDT) In-Reply-To: <20180405084300.GB31585@thebird.nl> (Pjotr Prins's message of "Thu, 5 Apr 2018 10:43:00 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Pjotr Prins writes: > I think we should have a switch for turning off tests. Let the builder > decide what is good or bad. Too much nannying serves no one. I think it would be OK to give users the choice of not running tests when building from source, if they really don't want to. This is similar to how users can choose to skip the "make check" step (and live with the risk) when building something manually. However, I think we should always run the tests by default. Maybe you could submit a patch to add a "--no-tests" option? ludo@gnu.org (Ludovic Court=C3=A8s) writes: > That is why I was suggesting putting effort in improving substitute > delivery rather than trying to come up with special mechanisms. Yes, I think that improving substitute availability is the best path forward. I'm willing to bet that Pjotr would not be so frustrated if substitutes were consistently available. Regarding Pjotr's suggestion to add a "test result substitute" feature: It isn't clear to me how a "test result substitute" is any better than a substitute in the usual sense. It sounds like Pjotr is arguing that if the substitute server can tell me that a package's tests have passed, then I don't need to run the tests a second time. But why would I have to build the package from source in that case, anyway? Assuming the substitute server has told me that the package's tests have passed, it is almost certainly the case that the package has been built and its substitute is currently available, so I don't have to build the package myself - I can just download the substitute! Conversely, if a substitute server says the tests have not passed, then certainly no substitute will be available, so I'll have to build it (and run the tests) myself. Perhaps I am missing something, but it does not seem to me that the existence of a "test result substitute" would add value. I think what Pjotr really wants is (1) better substitute availability, or (2) the option to skip tests when he has to build from source because substitutes are not available. I think (1) is the best goal, and (2) is a reasonable request in line with Guix's goal of giving control to the user. Ricardo Wurmus skribis: > An idea that came up on #guix several months ago was to separate the > building of packages from testing. Testing would be a continuation of > the build, like grafts could be envisioned as a continuation of the > build. What problems would that solve? Pjotr Prins writes: > The building takes long enough. Testing takes incredibly long with > many packages (especially language related) and are usually single > core (unlike the build). Eelco told me that in Nix, they set --max-jobs to the number of CPU cores, and --cores to 1, since lots of software has concurrency bugs that are easier to work around by building on a single core. Notably, Guix does the opposite: we set --max-jobs to 1 and --cores to the number of CPU cores. I wonder if you would see faster builds by adjusting these options for your use case? > It is also bad for our carbon foot print. Assuming everyone uses Guix > on the planet, is that where we want to end up? When everyone uses Guix on the planet, substitutes will be ubiquitous. You'll be able to skip the tests because, in practice, substitutes will always be available (which means an authorized substitute server ran the tests successfully). Or, if you are very concerned about correctness, you might STILL choose to build from source - AND run the tests - because you are concerned that your particular circumstances (kernel version, file system type, hardware, etc.) was not tested by the build farm. > I know there are two 'inputs' I am not accounting for: (1) hardware > variants and (2) the Linux kernel. But, honestly, I do not think we > are in the business of testing those. We can assume these work. Even if those components worked for the maintainers who ran the tests on their own machines and made a release, they might not work correctly in your own situation. Mark's story is a great example of this! For this reason, some people will still choose to build things from source themselves, even if substitutes are available from some other place. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAlrHNrwACgkQ3UCaFdgi Rp354Q//fDMiuWxLbX8YhYBRy3lg0NqKw3hLppWyn6+oo783UyYkEe0lJcNC+5sn F50lidLaLOQ5R6i3Pmlw7Y5cGs3gOXerp64s9ymw4yOx3GxHi+RhGtEuExrddOZ3 IRlZ5XBuwbbUigHnvg49LoCT5JQyg5Lv8dWMl+chveRuat36HSHH6O/djRFTlJX2 Cv6SnFLpQpkTNYbnDOStu1zAHZe5uaVRxIZ9nA4GPgvNy/I+z0QD/pV5gTVogVUj c9rK7Hi0w2+g4gxwYwbc5PihcD/z0ItDKCX3FlF3/6cuI+3BDJv4hD18CJY9hr8K FvbhZ4XVu+cflnlRiOtWNkaYWSvMG3UbBsyYxvG28eSxc4+0sa0J7lX0psCrKMPF 6xStZqwjMqx3G3s7J4H0IPA1MlxKDf1NJbOzK74AP7+OtcjnrdKuK5MOs4/VcEWe iri0Lw20FxbsqSbNzPSHT/dLsquWRBIC37SWVBLFDESgFA0K8Ig3qBHIr9oXQhSh QhBFNTe38njixwkyFxa3YIDsrziflTp3DRYp1/3zg92qletkz1CJ3Dc7dYVKLVEm tseAipt2/mF897qjer8dwRtTKs5r+jAKtwKXNk33fzni3iDadbE+1TU7zefbNT0k TSkuAMgtE2I54eeiKLxcnmnRKyT/KV0QcXOy58vScQZoNz424g4= =5QYE -----END PGP SIGNATURE----- --=-=-=--