From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id UMhZM5lhQWUjngAAG6o9tA:P1 (envelope-from ) for ; Tue, 31 Oct 2023 21:20:42 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id UMhZM5lhQWUjngAAG6o9tA (envelope-from ) for ; Tue, 31 Oct 2023 21:20:41 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C08D910536 for ; Tue, 31 Oct 2023 21:20:41 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="bL9E/ijU"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1698783641; a=rsa-sha256; cv=none; b=sUaS4AhkzVzQo8e8Iuil/MvQEYEwWgU0oCm5rPgDah+1ghTONbk7IcfwrKM37jz8EYFgvW 627/mMq79AwMldKWTMnQCTY0CmUY/EC+g/HJvgK4ut+YcJ1F9Qw55DzUgE6cpCbCPl6I1Q 25efaPOyaSzs0ssmbJthDwAfUKyDoCWYhjCjqXgQC7TWbNlbJz5xGSCv0OsWKkT38BKKV+ scOmZgcSbBNuUmpBKTETgnVfm8T0/tEpKV8sMwA97HYZRZUX1PVY72AAe6VJ1Dt4lgGBsJ RsTJwIfgKcTlv355ljWQGamXk5hLXQEdGndASG+iJOcb4R3WOznRy9krlJckAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1698783641; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=TmRKVmwSMQknOsUxqXg5ueg7yXnTQjju1pWQ9umztNQ=; b=uXOV1o/JeEMNVJvq8j0GdHeT8f/CGpaX6DPc57H8krRu0ZmXgKAbyPVSK6c08L+QkCW+IE ImqsmgVdi7ApMNKBBDK3hIdMTLHFnXAWvFZlN+qwFcY+dQ4aNPAaGyUYSjF/u9xhoa7RK1 SGOySXfI+YKK3hd5B/c3+N3Ij7SxiCruo/BdZFcPs2SDsB9r7zGi3fv+kbWw3Uu8ci36Rz HDRY0Ajv5GoYW5QDNlPrlQS7/a+zM8X+HOtdFes2AssPTDypG/HPdTECVhmiJVkFqQA5pp jfHaWHjqu8Jy8dc0+liF3cSrY4jP6/VYnub0Op9yUEVnY7Hl/opg+5OawNSzqw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="bL9E/ijU"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxvDo-0000FP-IU; Tue, 31 Oct 2023 16:20:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxvDi-0000Dq-Dl for guix-devel@gnu.org; Tue, 31 Oct 2023 16:20:15 -0400 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qxvDZ-0003hj-MB for guix-devel@gnu.org; Tue, 31 Oct 2023 16:20:12 -0400 Received: by mail-qk1-x732.google.com with SMTP id af79cd13be357-77891f362cfso17915185a.1 for ; Tue, 31 Oct 2023 13:20:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698783603; x=1699388403; darn=gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TmRKVmwSMQknOsUxqXg5ueg7yXnTQjju1pWQ9umztNQ=; b=bL9E/ijUE5D5zo9TGOnlqzokuuL8omIUBO+X/yClP60g5MPoQXPAONWblvniS4lSIY IxCV2uSN1wDokOjadGJuYgK1Yxf/7MSYmugA0GZcN0YxEV+DQWWIllU3gkSjEXi/daOL pbzzuV3YB8RudMqLbcE33MOCqxQ1Rsbd/9bjSSuo4CK0n/O3o3H4I/ewit0bLBkSGr2U IwMG/HFomMeYgU52pfo7uoLvC7KNoeK9XmGYt4BxTXBDl48njAvgczrjUp7bDcMMICpD QYrmpSCuHlDI4raEGuHozq9JkOhDBOUdwb+6mfYwd9zzu0u13AUAEN/biKVGzJCasFFH zOKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698783603; x=1699388403; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TmRKVmwSMQknOsUxqXg5ueg7yXnTQjju1pWQ9umztNQ=; b=Xs20goF/tsBIc/bVxBGJjZrzFH8UpLPTUOy+SQjIukfNvPewinznTONw3cGPudWOeS VrycTKgqr0i/tqbIO/f4D5ynKr+QNwgyZBOWayZb9Ie1IR2cqMAdTrf2ndgMW5LRCdlx LSEDaptZQACH12KTwRaA/7NfD9Egu3rjshPs1W/Qflbsl6YVWZFb+w0o1b+klH40HB+h KPpiLGo7iCIgvsVlRrVg8gi/BM9uGy7YYlEwFF0/jEiL264hkt9jBNaT2kMQkGo5FVOD W6BhHQ04b9RT8BTtbok4hnxIB+FaXA3ebpW0z6LaeS0Nq8Q1ULG+xIo/Dta6WM/UCOsY FgIA== X-Gm-Message-State: AOJu0YyGFqaQFe4W5FBWNZXfheYhBrDzNUWbFa097IEBYXRHpaOze3zK WispNnbdj7vkPjUW+PJEBxwIZQ58l4segQ== X-Google-Smtp-Source: AGHT+IG7rVa4PwlvuEAdtcr/FWlqJf8qbFDicwDzLTKrss8pYSMqlsEy7GcOaCL5dJddZnjdJtS/1g== X-Received: by 2002:a05:620a:2490:b0:779:d17f:68c8 with SMTP id i16-20020a05620a249000b00779d17f68c8mr1198484qkn.18.1698783603629; Tue, 31 Oct 2023 13:20:03 -0700 (PDT) Received: from hurd (dsl-10-128-239.b2b2c.ca. [72.10.128.239]) by smtp.gmail.com with ESMTPSA id g2-20020a05620a13c200b007788dac6b24sm809130qkl.41.2023.10.31.13.20.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 13:20:03 -0700 (PDT) From: Maxim Cournoyer To: Bruno Victal Cc: guix-devel@gnu.org Subject: Re: Expressing system test dependencies In-Reply-To: <87wmv9igyt.fsf@makinata.eu> (Bruno Victal's message of "Thu, 26 Oct 2023 15:41:30 +0100") References: <87wmv9igyt.fsf@makinata.eu> Date: Tue, 31 Oct 2023 16:20:02 -0400 Message-ID: <878r7ipmrx.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::732; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x732.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx11.migadu.com X-Spam-Score: -9.41 X-Migadu-Queue-Id: C08D910536 X-Migadu-Spam-Score: -9.41 X-TUID: V3o9YsZ6tPK7 Hi Bruno, Bruno Victal writes: > Hi, > > There are system tests that would benefit from being able to express > that their results are dependent on the result of other tests, when > these constitute independent units, especially when there are > configuration variants present. (e.g. gnu/tests/gdm.scm) > > Consider the following files/system-tests [1]: > * gnu/tests/gdm.scm > Right now this module implements two tests: %test-gdm-x11 and > %test-gdm-wayland. (whose names are self-explanatory) > > * gnu/tests/vnc.scm > This module implements one test, %test-xvnc, in particular it does so > by testing the XDMCP feature using GDM. > Note: this module predates gnu/tests/gdm.scm. > > Here's where we would benefit from being able to express system-test > dependencies. Clearly some of the =E2=80=9Ctest-assert=E2=80=9D clauses in > gnu/tests/vnc.scm would be better placed in gnu/tests/gdm.scm as they > are GDM specific tests. Since they constitute configuration variants of > the service, it would be placed into a %test-gdm-autosuspend variable. > > The rationale for a test-dependency mechanism is as follows: > Suppose we split the GDM specific test to (gnu tests gdm). Now let's say > that we run %test-xvnc and it fails. Is the failure due to GDM or is it > caused by something else? (within the %test-xvnc) > > By splitting the test to (gnu tests gdm) we would have to run the GDM > tests first, which isn't obvious. If we could express a dependency here, > the debugging experience is improved as we now have a way to know that a > integral component involved in the test failed. (or could be used to > rule out other parts) > > Since the system-test results are expressed as derivations, successful > tests shouldn't result in duplicate runs so perhaps we could make use of > this fact for the effect? Ideally, any kind of tests would be independent from each other, but for system tests where it's an integration of various components, we do rely on the individual components used in a test working. I'm not sure it's worth adding more complexity though; a failure in GDM should have been caught at the time packaging/upgrading GDM with its own test suite; it seems these cases would be very rare, and we have actual real problems to fix (c.f. our bug tracker) :-). --=20 Thanks, Maxim