From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Taylan Kammer Newsgroups: gmane.lisp.guile.bugs Subject: bug#72384: srfi-64: test-end should not clear fail list Date: Wed, 2 Oct 2024 01:38:33 +0200 Message-ID: <7c410ef8-8c73-4107-940e-dba14273d229@gmail.com> References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="------------cJzl5h4QALw3IDzoIWWmRazb" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30443"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird To: Tomas Volf <~@wolfsden.cz>, 72384@debbugs.gnu.org Original-X-From: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Wed Oct 02 01:40:16 2024 Return-path: Envelope-to: guile-bugs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1svmTX-0007kb-JO for guile-bugs@m.gmane-mx.org; Wed, 02 Oct 2024 01:40:15 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svmTN-0005oM-CB; Tue, 01 Oct 2024 19:40:05 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svmTL-0005nY-Ch for bug-guile@gnu.org; Tue, 01 Oct 2024 19:40:03 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1svmTK-0002b2-CE for bug-guile@gnu.org; Tue, 01 Oct 2024 19:40:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=In-Reply-To:From:References:MIME-Version:Date:To:Subject; bh=vHe1jSfXzu4vfm/jjNHhVnXPEvrALau8cEjhD3qmQx8=; b=SrgSeiNUH2q1Z/n+1PQWfGX7w/kfPWa5DwOT/8yyL83mYDSAAUb+R3knF5jhSS5YXsazZFZk+iEbnMiDBTCS1V7vNa8Lt3tp/Ewq1bBBVIyjramLNemWXoiKOOPgzWZ4NksUmXenQIKuFDyDogQBNcY3gqqRlxSBt5ydADGGIfuL9QQoSZcZponi+d0GKhZ6vRiUswOmXVk/a0V52gUFIis/M6eTlyr5LLP3d9mU67ynz2HolTmwkBBEU0MOq13AAwiz7KSGQ8IsME1WASBdtdGXgNu5q2oeeub6101RfiUlEXx1uopTc2r6+HCT5lWinL9WZ87A+OGguScAhZciww==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1svmTJ-0000Wu-VT for bug-guile@gnu.org; Tue, 01 Oct 2024 19:40:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Taylan Kammer Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Tue, 01 Oct 2024 23:40:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 72384 X-GNU-PR-Package: guile Original-Received: via spool by 72384-submit@debbugs.gnu.org id=B72384.17278259792020 (code B ref 72384); Tue, 01 Oct 2024 23:40:01 +0000 Original-Received: (at 72384) by debbugs.gnu.org; 1 Oct 2024 23:39:39 +0000 Original-Received: from localhost ([127.0.0.1]:54391 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1svmSw-0000WW-GZ for submit@debbugs.gnu.org; Tue, 01 Oct 2024 19:39:39 -0400 Original-Received: from mail-wm1-f41.google.com ([209.85.128.41]:39344) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1svmSu-0000WQ-NC for 72384@debbugs.gnu.org; Tue, 01 Oct 2024 19:39:37 -0400 Original-Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-42cddc969daso8916815e9.1 for <72384@debbugs.gnu.org>; Tue, 01 Oct 2024 16:39:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727825916; x=1728430716; darn=debbugs.gnu.org; h=in-reply-to:from:content-language:references:to:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=vHe1jSfXzu4vfm/jjNHhVnXPEvrALau8cEjhD3qmQx8=; b=JeYkX7cuZ3umWe4e3emqLg0JUdJ7/Y44DtAQAbCfekPdyKdx27C5a09AkAVN6mrxm6 Ji6oAlFgDfny5ovJmuNokub1UCBiB7iKKZ86mnxpwgcfle0cWRP+HcEjkX0mQ5egbDJp P13Ps1CWtZxKzS3XyyxtiQC3dQ2hhy2FGJZAIp2ElsAzVq8ci3k/uUhg0TukAj7SDfs8 Hz7+5VO8byWqEvof6X6VUNzTr/cYp1ZZuXen0EGgF076tFuwsVR4RGv2+swRwxDS6l6l GPS2j2Ttnw0yVmFzP+iiPVE6VcUt/QWOdEO12Jda8Ega3GeWTBejNTHuIougXpoXcDrV dQiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727825916; x=1728430716; h=in-reply-to:from:content-language:references:to:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=vHe1jSfXzu4vfm/jjNHhVnXPEvrALau8cEjhD3qmQx8=; b=FeMzAC204kVc3OZVD6qCREbVB7nD9zvGPeoV5qYZCPMSFV6vE9hwbmGRQd1WFFjI4T EDW1JUn3Lx4+C2IMwZE8AaFQxGXZse43lJwMOYOqAKgfa5hyBHaa1iGELy1RvgQbUV2S L7h6iComCHuhY0vHEmN9shr7QP3hCU360o3m9i8WEh3N8gLXma6qMXe1/z+hbXgluX5k GlLA6ncL7KVFIRwq34Ss8z5ipsKtmPU2ZukrmQ+BUueqrnXqT/zEckSIqT8jU8ySpVAt MhHaH1qCnSG6uwF5tR0YgC8ywwX4wXB37Lof9MDftTDGiC3fgyXT4sSYSTALp/VXB0AN ISGw== X-Forwarded-Encrypted: i=1; AJvYcCVxQsUt6Fj6FVhTmQn0KpaBn2dt9XmhqJiHnLXqI/Uvz6wXEMUAZ0itrOCPTHQ0ZRg0d+Iksw==@debbugs.gnu.org X-Gm-Message-State: AOJu0YxsgPSG+IqNPnQbHDzkrdzqKhnbPIOuSZ2PfDIjpr9DoeIquf2q IFNwnnyGKgF1POo846j2qAdPEn4cL7y+jAMV9BZ7yIJcGJ3qFW24V2QxZ3NZsnw= X-Google-Smtp-Source: AGHT+IGTEi8t00wLNfcM5nNccwv/y6YQqiIRtu5KvRCFd1MtkNMSEwsKGMcKkWvCdSDZbXOrS3FKAA== X-Received: by 2002:adf:f844:0:b0:37c:d0d6:ab1a with SMTP id ffacd0b85a97d-37cfba19cd5mr299540f8f.12.1727825915995; Tue, 01 Oct 2024 16:38:35 -0700 (PDT) Original-Received: from ?IPV6:2003:106:8f04:c300:95ac:529d:6db3:196b? (p200301068f04c30095ac529d6db3196b.dip0.t-ipconnect.de. [2003:106:8f04:c300:95ac:529d:6db3:196b]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c88245e763sm6821226a12.50.2024.10.01.16.38.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 01 Oct 2024 16:38:34 -0700 (PDT) Content-Language: en-US In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Original-Sender: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.bugs:11016 Archived-At: This is a multi-part message in MIME format. --------------cJzl5h4QALw3IDzoIWWmRazb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 30.07.2024 21:52, Tomas Volf wrote: > Hello, > > I think I found a bug in (srfi srfi-64) module shipped with GNU Guile. > > Reading the specification for test-expect-fail I do not see a mandate to clear > expect-fail list on test-end. test-skip does have such provision, but it is > lacking in the test-expect-fail. Therefore I think current behavior is wrong: > > (use-modules (srfi srfi-64)) > > (test-begin "x") > > (test-begin "group1") > (test-expect-fail "test-a") > (test-assert "test-a" #t) > (pk (test-result-kind)) > (test-end "group1") > (test-assert "test-a" #t) > (pk (test-result-kind)) > > (test-end) > > Leading to: > > ;;; (xpass) > > ;;; (pass) > > Have a nice day > Tomas Volf > > I'm inclined to see this as an error/omission in the spec itself. It makes sense for test-end to clear the expected-fail list, just like it clears the skip list. If it didn't, it might cause one to accidentally mark tests as "expected failure" that weren't meant to be marked as such. Consider the following: (test-group "group1" (test-expect-fail "test-a") (test-assert "test-a" #t)) (test-group "group2" (test-assert "test-a" #t)) Since test-group is equivalent to a pair of test-begin/test-end calls, this code would be very "deceptive" if the implicit test-end didn't clear the expected-fail list. After all, the two groups look completely disjoint, and one wouldn't expect any state from the former to implicitly bleed into the latter. Also, if I'm not mistaken, there's not even a way to clear the expected-fail list explicitly. I actually have some "real-world" code that uses repetitive names in a test suite within different groups, so this isn't just a theoretical issue either:     https://codeberg.org/taylan/scheme-bytestructures/src/branch/master/run-tests.body.scm Notice how often the names "ref" and "set" are used. So, I think the behavior of the reference implementation is correct/desirable here. My implementation of SRFI-64 does the same. - Taylan --------------cJzl5h4QALw3IDzoIWWmRazb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 30.07.2024 21:52, Tomas Volf wrote:
Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

Reading the specification for test-expect-fail I do not see a mandate to clear
expect-fail list on test-end.  test-skip does have such provision, but it is
lacking in the test-expect-fail.  Therefore I think current behavior is wrong:

    (use-modules (srfi srfi-64))

    (test-begin "x")

    (test-begin "group1")
    (test-expect-fail "test-a")
    (test-assert "test-a" #t)
    (pk (test-result-kind))
    (test-end "group1")
    (test-assert "test-a" #t)
    (pk (test-result-kind))

    (test-end)

Leading to:

    ;;; (xpass)

    ;;; (pass)

Have a nice day
Tomas Volf

I'm inclined to see this as an error/omission in the spec itself. It makes sense for test-end to clear the expected-fail list, just like it clears the skip list. If it didn't, it might cause one to accidentally mark tests as "expected failure" that weren't meant to be marked as such. Consider the following:

(test-group "group1"
  (test-expect-fail "test-a")
  (test-assert "test-a" #t))

(test-group "group2"
  (test-assert "test-a" #t))

Since test-group is equivalent to a pair of test-begin/test-end calls, this code would be very "deceptive" if the implicit test-end didn't clear the expected-fail list. After all, the two groups look completely disjoint, and one wouldn't expect any state from the former to implicitly bleed into the latter. Also, if I'm not mistaken, there's not even a way to clear the expected-fail list explicitly.

I actually have some "real-world" code that uses repetitive names in a test suite within different groups, so this isn't just a theoretical issue either:

    https://codeberg.org/taylan/scheme-bytestructures/src/branch/master/run-tests.body.scm

Notice how often the names "ref" and "set" are used.

So, I think the behavior of the reference implementation is correct/desirable here. My implementation of SRFI-64 does the same.

- Taylan

--------------cJzl5h4QALw3IDzoIWWmRazb--