From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Taylan Kammer Newsgroups: gmane.lisp.guile.bugs Subject: bug#72367: srfi-64: nested group not counted as one test Date: Wed, 2 Oct 2024 06:32:23 +0200 Message-ID: <3bf34f4a-6ed2-46cf-a31e-108b1dcab1ec@gmail.com> References: <36bb1a5e-b212-4a1d-88dc-c88e16a47182@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="40897"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird To: Tomas Volf <~@wolfsden.cz>, 72367@debbugs.gnu.org Original-X-From: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Wed Oct 02 08:25:21 2024 Return-path: Envelope-to: guile-bugs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1svsnZ-000AS8-21 for guile-bugs@m.gmane-mx.org; Wed, 02 Oct 2024 08:25:21 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svsnI-0000Al-Gz; Wed, 02 Oct 2024 02:25:04 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svsnG-0000AA-Ig for bug-guile@gnu.org; Wed, 02 Oct 2024 02:25:02 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1svsnG-000088-9h for bug-guile@gnu.org; Wed, 02 Oct 2024 02:25:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=In-Reply-To:References:From:MIME-Version:Date:To:Subject; bh=EyVXO/BYkkw4kG5X93/k70LvNgSh5V3SprAdpY3Xx60=; b=UL2Ch7W9FCKdZhCB9B3/DsmmcZULcLbQCblW7U+iOP9LAVZVc43zOwWfw6THxiwPofswwqTFuB9awvoi+lwebRFmKaWmk/ZlhYz5p1m8YVnHPgH30Sxssah4keQILOqV1aDRB0argzNHzSkvHCLJoZ7j28otLnJ5JQMyGA3uhCoDPp+VUP0eMdLGl/yWdBRH25ypEswzzfmZdP6mKvA+PRvNoBKTcHQYjo6Cz+umNbk4FrqMg4v4rT/Uv6Sp4qhk1pGl9MSfITRiyrraOHksR2ySepm6SVAleDqeBFByQRgCwEHJpYgbt/HQFQUkRLCtG61JDDrQt4d9Yait0LHEjw==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1svsnG-0002Yr-9X for bug-guile@gnu.org; Wed, 02 Oct 2024 02:25:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Taylan Kammer Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Wed, 02 Oct 2024 06:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 72367 X-GNU-PR-Package: guile Original-Received: via spool by 72367-submit@debbugs.gnu.org id=B72367.17278502889816 (code B ref 72367); Wed, 02 Oct 2024 06:25:02 +0000 Original-Received: (at 72367) by debbugs.gnu.org; 2 Oct 2024 06:24:48 +0000 Original-Received: from localhost ([127.0.0.1]:56336 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1svsn1-0002YF-MU for submit@debbugs.gnu.org; Wed, 02 Oct 2024 02:24:48 -0400 Original-Received: from mail-ej1-f47.google.com ([209.85.218.47]:41955) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1svsn0-0002Y7-2h for 72367@debbugs.gnu.org; Wed, 02 Oct 2024 02:24:46 -0400 Original-Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-a8a7dddd2c3so72229266b.3 for <72367@debbugs.gnu.org>; Tue, 01 Oct 2024 23:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727850224; x=1728455024; darn=debbugs.gnu.org; h=content-transfer-encoding:in-reply-to:content-language:references :to:from:subject:user-agent:mime-version:date:message-id:from:to:cc :subject:date:message-id:reply-to; bh=EyVXO/BYkkw4kG5X93/k70LvNgSh5V3SprAdpY3Xx60=; b=McwFxcabp2Q2dx4j/2UO74suSN3LCbP41FGRcGUG+25bdERb6A24ArfQiHJWMmb/gU WsU4hbkRxUaKlJzYPCeebSv0THSRbUyzIhUEmZsOdLVnk9gcX7ZvR8PoJINU/M5RUkEU ew9D43zRGmbWuUIr5w2HGjMFe1En4zYvuL3z3ITVk12xXzWezhF56IJE8vnPF/hrV9C9 LrR08i7i+V6HJw11Jj+WtVluaRGG+6sLO0EeYVL5iUwZHOCX+r3f8oEbY/mC2H9JGDMI tw+eGTj8uVjzhMGZl2iWhpgBySYU55GYuhvzcbVBXDX7YWVEIECtZITmlUIusBU47CW4 cHEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727850224; x=1728455024; h=content-transfer-encoding:in-reply-to:content-language:references :to:from:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EyVXO/BYkkw4kG5X93/k70LvNgSh5V3SprAdpY3Xx60=; b=l9jB/He5uKfK1y7S8pDphQwPUxlxMf3Cd+lxhH5GQA4HyVF26k5G7Z/V68mA7+hzIs eCuEfP1r/jR14otjDKXhoETa3Nu7jJoAIVs1h8saOEwy3AtSzVBeIrgYI+PnCDmmpyT9 UxnCBIL3uTIkIKuO8GXc1vRxlzaP+KyrSwJYdwjufvVg/UjPDNDCHTRL+N4NaDwURoSw z6XymY3HTtkIQeoCqrjOusXglC/8RMlq5XST6TgsFvPhNOMBc8Z2Ddy8jS/UW36yYKMV WZ31sVnqS0gVTgrV3pizNhZ/huHIi2fFIZWa8yByQu7KSagHN5tS9LRHd1iVqjkKGqn0 S6DA== X-Forwarded-Encrypted: i=1; AJvYcCVwV11kNlaO2kcgyCpWL0vyyXdK/6jDUFc81qtMfwGUIusjBcPHqTwxooE+6M4CPJfaYr952A==@debbugs.gnu.org X-Gm-Message-State: AOJu0Yx+xWyT0CvN9z8p3BOhtyDoQsxPeIctExNY6ZRbRkWDXFY+bQhe Xj5nHD67OdxpPqoSk10tnxO4M9YPHra+3ISBjPlZeU+dK6VGM17Hmnut/9+KL0Y= X-Google-Smtp-Source: AGHT+IG0dZD0174r40nxW2unr8sF1HPPQWP9neAoKEy1kQbw57SmWTxcvrWudo5vFreRG3NY2mMS0g== X-Received: by 2002:a05:600c:45cc:b0:42c:df54:18f6 with SMTP id 5b1f17b1804b1-42f777b7936mr5966535e9.3.1727843546643; Tue, 01 Oct 2024 21:32:26 -0700 (PDT) Original-Received: from ?IPV6:2003:106:8f04:c300:95ac:529d:6db3:196b? (p200301068f04c30095ac529d6db3196b.dip0.t-ipconnect.de. [2003:106:8f04:c300:95ac:529d:6db3:196b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42f79ead1absm7132875e9.17.2024.10.01.21.32.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 01 Oct 2024 21:32:26 -0700 (PDT) Content-Language: en-US In-Reply-To: <36bb1a5e-b212-4a1d-88dc-c88e16a47182@gmail.com> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Original-Sender: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.bugs:11018 Archived-At: On 30.09.2024 16:42, Taylan Kammer wrote: > On 30.07.2024 21:51, Tomas Volf wrote: >> Hello, >> >> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile. >> >> The specification says the following about the count argument to the test-begin >> form: >> >>> The optional count must match the number of test-cases executed by this >>> group. (Nested test groups count as a single test case for this count.) >> Thus I believe that following should *not* call on-bad-count-function. >> >> (use-modules (srfi srfi-64)) >> (let ((r (test-runner-null))) >> (test-runner-current r) >> (test-runner-on-bad-count! r (λ (runner actual-count expected-count) >> (pk (test-runner-group-stack runner)) >> (pk actual-count expected-count))) >> (test-begin "x" 1) >> (test-begin "y" 3) >> (test-assert #t) >> (test-assert #t) >> (test-assert #t) >> (test-end) >> (test-end)) >> >> However as we can see it does: >> >> ;;; (("x")) >> >> ;;; (3 1) >> >> Have a nice day >> Tomas Volf > Looks like a bug indeed. I believe I was able to fix it in my implementation with this commit: > >     https://codeberg.org/taylan/scheme-srfis/commit/c49e367fbf292c6ee66ff435c7daa1f4ae5a47fa > > An equivalent fix could be applied to the upstream reference implementation / current Guile implementation. > I've also discovered another pair of bugs related to nested test groups, but only when using `test-group`. 1. Skipping a `test-group` by name was not working. The "meta test suite" of SRFI-64 had an associated test case marked as an expected failure with the comment "???". I've found why it wasn't working, and fixed it. 2. Had the skipping actually worked, the code would have failed to increment the skip-count and total-count. I've made sure these are incremented appropriately when a `test-group` is skipped. One quirk remains, but is difficult to fix without substantial changes: A skipped `test-group` will not invoke any handlers whatsoever, so the runner can't detect this case to write something into the log. Normally, a skipped test would cause something to be logged in my implementation, but a skipped `test-group` will be skipped silently and only appear as a statistic in the summary of the counts at the end. Fixing this would require "privileging" my `test-runner-simple` implementation by giving it access to undocumented APIs, or explicitly expanding the SRFI-64 API. Another option would be to slightly change the semantics of `test-group`, such that the on-group-begin and on-group-end handlers are always invoked, and only the body of the group skipped if the group is supposed to be skipped. This is how skipping of individual tests is handled: The on-test-begin and on-test-end handlers *are* called for skipped tests. So, this change would make the API more coherent. It would probably also simplify the implementation. The downside is that it would be a clear contradiction of the spec. It seems like skipping of test groups was kind of an afterthought and wasn't integrated cleanly with the rest of SRFI-64. Opinions welcome on whether I should simply make my implementation break off from the spec here. I imagine there isn't much code out there that strictly requires group-begin/group-end to *not* be called in case of a `test-group` that's meant to be skipped. - Taylan