T350-crypto T357-index-decryption: possible race condition?

T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

Hi there,

my regular notmuch test builds recently started to fail, more
concretely: the test suite fails because some subtests are KILLed.
Building notmuch 0.37 with my usual spec-file as a rawhide-mock build
(a local chroot for the development "version" of Fedora which will
become Fedora 39) I see:
```
T350-crypto: Testing PGP/MIME signature verification and decryption
 PASS   emacs delivery of signed message via fcc
 PASS   emacs delivery of signed message via fcc and smtp
 PASS   signed part content-type indexing
 PASS   signature verification
 PASS   detection of modified signed contents
 PASS   corrupted pgp/mime signature
 PASS   signature verification without full user ID validity
 PASS   signature verification with signer key unavailable
```
There the suite "hangs" for about 2 minutes, followed by
```
FATAL: /builddir/build/BUILD/notmuch-0.37/test/T350-crypto.sh:
interrupted by signal 15
```
It proceeds until
```
T357-index-decryption: Testing indexing decrypted mail
```
and hangs again for about 2 minutes, followed by
```
FATAL: /builddir/build/BUILD/notmuch-0.37/test/T357-index-decryption.sh:
interrupted by signal 15
```
In the end, the suite complains:
```
'/builddir/build/BUILD/notmuch-0.37/test/test-results/T350-crypto'
does not exist!
'/builddir/build/BUILD/notmuch-0.37/test/test-results/T357-index-decryption'
does not exist!
```
At least for T350 this is strange because several subtests ran and
passed! This indicates a race or a wrong signal trap.

The same problem happens with notmuch 0.37 in Fedora's infrastructure
(koji rawhide, e.g.
https://koji.fedoraproject.org/koji/taskinfo?taskID=101014703).

Curiously, everything seems to work with notmuch 0.37 in Fedora 38,
which is the current release, in both koji and locally in mock.

BUT: In Fedora's secondary test-bed (copr) and with notmuch from git,
these kind of errors happen on released fedora versions, too. This was
kind of erratic, but I suspected something related to emacs 28 and
test timeouts. So I increased the timeout in the test lisp lib (see
below), hoping for the better, but getting the worse, at least
deterministically worse: With this change, the test suite fails
reliably (the two mentioned above plus T315-emacs-tagging) on all
Fedoras (with Emacs 28) and passes on epel (with Emacs 27), see for
example:
https://copr.fedorainfracloud.org/coprs/mjg/notmuch-git/build/5908525/

Now, emacs is not the only difference, and the complete test result
directory disappearing is still strange, and really all that is
strange. Help please ;)

(There is another problem related to Python 3.12 which I'll address
separately - rawhide still carries 3.11.)

```
---
 test/test-lib.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/test-lib.el b/test/test-lib.el
index 79a9d4d6..39ade9b9 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -39,7 +39,7 @@
 (defun notmuch-test-wait ()
   "Wait for process completion."
   (while (get-buffer-process (current-buffer))
-    (accept-process-output nil 0.1)))
+    (accept-process-output nil 120)))

 (defun test-output (&optional filename)
   "Save current buffer to file FILENAME.  Default FILENAME is OUTPUT."
```

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:

> Building notmuch 0.37 with my usual spec-file as a rawhide-mock build
> (a local chroot for the development "version" of Fedora which will
> become Fedora 39) I see:
> ```
> T350-crypto: Testing PGP/MIME signature verification and decryption
>  PASS   emacs delivery of signed message via fcc
>  PASS   emacs delivery of signed message via fcc and smtp
>  PASS   signed part content-type indexing
>  PASS   signature verification
>  PASS   detection of modified signed contents
>  PASS   corrupted pgp/mime signature
>  PASS   signature verification without full user ID validity
>  PASS   signature verification with signer key unavailable
> ```
> There the suite "hangs" for about 2 minutes, followed by

This sounds suspiciously like "timeout" kicking in and killing a test
that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the
environment to some smaller/larger number to test this.

The first subtest in T357 is also sending an encrypted message, so it
looks like some bad interaction between gpg and emacs. Maybe you can try
sending an encrypted message from emacs interactively in your chroot
environment.

0) you will first need to run 

   gpg --no-tty --import ./test/openpgp4-secret-key.asc 

this will create a keyring etc... in the chroot

1) You can use the script

./devel/try-emacs-mua -q


2) Copy the following into *scratch* and run with C-j after the last
closing paren (this is just copied from the test suite internals)

(let ((message-send-mail-function (lambda () t))
      (mail-host-address "example.com"))
  (notmuch-mua-mail)
  (message-goto-to)
  (insert "test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000")
  (message-goto-subject)
  (insert "My Subject")
  (message-goto-body)
  (insert "a body")
  (mml-secure-message-encrypt)
  (let ((mml-secure-smime-sign-with-sender t)
	(mml-secure-openpgp-sign-with-sender t))
    (notmuch-mua-send-and-exit)))

You will probably need to answer "c" to create the sent-mail folder at
the prompt.

> ```
> In the end, the suite complains:
> ```
> '/builddir/build/BUILD/notmuch-0.37/test/test-results/T350-crypto'
> does not exist!
> '/builddir/build/BUILD/notmuch-0.37/test/test-results/T357-index-decryption'
> does not exist!
> ```
> At least for T350 this is strange because several subtests ran and
> passed! This indicates a race or a wrong signal trap.

Note that those files are summaries created by test_done, so it's not
that surprising that they are not there, since test_done is not reached
in those files.

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

Am Do., 11. Mai 2023 um 22:49 Uhr schrieb David Bremner <david@tethera.net>:
>
> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>
...
> > T350-crypto: Testing PGP/MIME signature verification and decryption
...
> >  PASS   signature verification with signer key unavailable
> > ```
> > There the suite "hangs" for about 2 minutes, followed by
>
> This sounds suspiciously like "timeout" kicking in and killing a test
> that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the
> environment to some smaller/larger number to test this.

Yes, when I set the timeout to 10m it hangs for 10min there,

> The first subtest in T357 is also sending an encrypted message, so it

Same after that one. I"ll attach "hang1" and "hang2" which are the
process lists during those hangs. Are those subtests the only ones (or
rather, the first subtests in these tests, and those tests the only
ones) sending encrypted messages?

> looks like some bad interaction between gpg and emacs. Maybe you can try
> sending an encrypted message from emacs interactively in your chroot
> environment.
>
> 0) you will first need to run
>
>    gpg --no-tty --import ./test/openpgp4-secret-key.asc
>
> this will create a keyring etc... in the chroot
>
> 1) You can use the script
>
> ./devel/try-emacs-mua -q
>
>
> 2) Copy the following into *scratch* and run with C-j after the last
> closing paren (this is just copied from the test suite internals)
>
> (let ((message-send-mail-function (lambda () t))
>       (mail-host-address "example.com"))
>   (notmuch-mua-mail)
>   (message-goto-to)
>   (insert "test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000")
>   (message-goto-subject)
>   (insert "My Subject")
>   (message-goto-body)
>   (insert "a body")
>   (mml-secure-message-encrypt)
>   (let ((mml-secure-smime-sign-with-sender t)
>         (mml-secure-openpgp-sign-with-sender t))
>     (notmuch-mua-send-and-exit)))
>
> You will probably need to answer "c" to create the sent-mail folder at
> the prompt.

Thanks for these instructions. I don't get a prompt: Right after C-j,
that emacs session hangs indefinitely. There was also something about
"require nnheader" or something like that. "hang3" is the hanging
process, which seems to be the same as for the other runs.

> > At least for T350 this is strange because several subtests ran and
> > passed! This indicates a race or a wrong signal trap.
>
> Note that those files are summaries created by test_done, so it's not
> that surprising that they are not there, since test_done is not reached
> in those files.

I assumed the subtests would timeout separately and the test framework
would catch that. If the test script itself is killed then yes, that
outcome is expected :)

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

[-- Attachment #1: Type: text/plain, Size: 3009 bytes --]

oh well, attachments ...

Am Fr., 12. Mai 2023 um 19:42 Uhr schrieb Michael J Gruber
<michaeljgruber+grubix+git@gmail.com>:
>
> Am Do., 11. Mai 2023 um 22:49 Uhr schrieb David Bremner <david@tethera.net>:
> >
> > Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
> >
> ...
> > > T350-crypto: Testing PGP/MIME signature verification and decryption
> ...
> > >  PASS   signature verification with signer key unavailable
> > > ```
> > > There the suite "hangs" for about 2 minutes, followed by
> >
> > This sounds suspiciously like "timeout" kicking in and killing a test
> > that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the
> > environment to some smaller/larger number to test this.
>
> Yes, when I set the timeout to 10m it hangs for 10min there,
>
> > The first subtest in T357 is also sending an encrypted message, so it
>
> Same after that one. I"ll attach "hang1" and "hang2" which are the
> process lists during those hangs. Are those subtests the only ones (or
> rather, the first subtests in these tests, and those tests the only
> ones) sending encrypted messages?
>
> > looks like some bad interaction between gpg and emacs. Maybe you can try
> > sending an encrypted message from emacs interactively in your chroot
> > environment.
> >
> > 0) you will first need to run
> >
> >    gpg --no-tty --import ./test/openpgp4-secret-key.asc
> >
> > this will create a keyring etc... in the chroot
> >
> > 1) You can use the script
> >
> > ./devel/try-emacs-mua -q
> >
> >
> > 2) Copy the following into *scratch* and run with C-j after the last
> > closing paren (this is just copied from the test suite internals)
> >
> > (let ((message-send-mail-function (lambda () t))
> >       (mail-host-address "example.com"))
> >   (notmuch-mua-mail)
> >   (message-goto-to)
> >   (insert "test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000")
> >   (message-goto-subject)
> >   (insert "My Subject")
> >   (message-goto-body)
> >   (insert "a body")
> >   (mml-secure-message-encrypt)
> >   (let ((mml-secure-smime-sign-with-sender t)
> >         (mml-secure-openpgp-sign-with-sender t))
> >     (notmuch-mua-send-and-exit)))
> >
> > You will probably need to answer "c" to create the sent-mail folder at
> > the prompt.
>
> Thanks for these instructions. I don't get a prompt: Right after C-j,
> that emacs session hangs indefinitely. There was also something about
> "require nnheader" or something like that. "hang3" is the hanging
> process, which seems to be the same as for the other runs.
>
> > > At least for T350 this is strange because several subtests ran and
> > > passed! This indicates a race or a wrong signal trap.
> >
> > Note that those files are summaries created by test_done, so it's not
> > that surprising that they are not there, since test_done is not reached
> > in those files.
>
> I assumed the subtests would timeout separately and the test framework
> would catch that. If the test script itself is killed then yes, that
> outcome is expected :)

[-- Attachment #2: hang1 --]
[-- Type: application/octet-stream, Size: 2406 bytes --]

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mjg       671258  0.0  0.0   5132  3712 ?        S    17:26   0:00 bash /builddir/build/BUILD/notmuch-0.37/test/notmuch-test
mjg       682895  0.0  0.0   2872  1664 ?        S    17:27   0:00 timeout 10m /builddir/build/BUILD/notmuch-0.37/test/T350-crypto.sh
mjg       682896  0.0  0.0   5228  3712 ?        S    17:27   0:00 bash /builddir/build/BUILD/notmuch-0.37/test/T350-crypto.sh
mjg       682913  0.0  0.0 225980  2976 ?        Ssl  17:27   0:00 gpg-agent --homedir /tmp/notmuch-test-13949.gpzozW/gnupg --use-standard-socket --daemon
mjg       682915  0.0  0.0  78828  2816 ?        SLl  17:27   0:00 scdaemon --multi-server --homedir /tmp/notmuch-test-13949.gpzozW/gnupg
mjg       682924  0.0  0.0   2728   512 ?        Ss   17:27   0:00 dtach -n /tmp/notmuch-test-13949.gpzozW/emacs-dtach-socket.13949 sh -c stty rows 24 cols 80; exec '/builddir/build/BUILD/notmuch-0.37/test/tmp.T350-crypto/run_emacs' ????--no-window-system ???? ????--eval '(setq server-name "notmuch-test-suite-13949")' ????--eval '(server-start)' ????--eval '(orphan-watchdog 13949)'
mjg       682925  0.2  0.5 302884 93568 pts/0    Ss+  17:27   0:01 emacs --quick --directory /builddir/build/BUILD/notmuch-0.37/emacs --load notmuch.el --directory /builddir/build/BUILD/notmuch-0.37/test --load test-lib.el --no-window-system --eval (setq server-name "notmuch-test-suite-13949") --eval (server-start) --eval (orphan-watchdog 13949)
mjg       683248  0.0  0.0   2868  1536 ?        S    17:27   0:00 emacsclient --socket-name=notmuch-test-suite-13949 --eval (notmuch-test-progn (let ((message-send-mail-function (lambda () t)) ?       (mail-host-address "example.com")) ?   (notmuch-mua-mail) ?   (message-goto-to) ?   (insert "test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000") ?   (message-goto-subject) ?   (insert "test encrypted message 001") ?   (message-goto-body) ?   (insert "This is a test encrypted message.\n") ?   (mml-attach-file "TESTATTACHMENT") (mml-secure-message-encrypt) ?   (let ((mml-secure-smime-sign-with-sender t) ?? (mml-secure-openpgp-sign-with-sender t)) ?     (notmuch-mua-send-and-exit))))
mjg       683254  0.0  0.0   8712  4736 ?        SLs  17:27   0:00 /usr/bin/gpg2 --no-tty --status-fd 1 --yes --command-fd 0 --armor --textmode --output /tmp/epg-outputeh9VEP --always-trust --encrypt -r 6D92612D94E46381

[-- Attachment #3: hang3 --]
[-- Type: application/octet-stream, Size: 153 bytes --]

/usr/bin/gpg2 --no-tty --status-fd 1 --yes --command-fd 0 --armor --textmode --output /tmp/epg-outputNffhxy --always-trust --encrypt -r 6D92612D94E46381

[-- Attachment #4: hang2 --]
[-- Type: application/octet-stream, Size: 2310 bytes --]

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mjg       694433  0.0  0.0   2872  1664 ?        S    17:37   0:00 timeout 10m /builddir/build/BUILD/notmuch-0.37/test/T357-index-decryption.sh
mjg       694434  0.0  0.0   5128  3712 ?        S    17:37   0:00 bash /builddir/build/BUILD/notmuch-0.37/test/T357-index-decryption.sh
mjg       694451  0.0  0.0 152248  3104 ?        Ssl  17:37   0:00 gpg-agent --homedir /tmp/notmuch-test-15324.9JGm7A/gnupg --use-standard-socket --daemon
mjg       694453  0.0  0.0  78828  2816 ?        SLl  17:37   0:00 scdaemon --multi-server --homedir /tmp/notmuch-test-15324.9JGm7A/gnupg
mjg       694462  0.0  0.0   2728   512 ?        Ss   17:37   0:00 dtach -n /tmp/notmuch-test-15324.9JGm7A/emacs-dtach-socket.15324 sh -c stty rows 24 cols 80; exec '/builddir/build/BUILD/notmuch-0.37/test/tmp.T357-index-decryption/run_emacs' ????--no-window-system ???? ????--eval '(setq server-name "notmuch-test-suite-15324")' ????--eval '(server-start)' ????--eval '(orphan-watchdog 15324)'
mjg       694463  0.7  0.5 302276 93472 pts/0    Ss+  17:37   0:00 emacs --quick --directory /builddir/build/BUILD/notmuch-0.37/emacs --load notmuch.el --directory /builddir/build/BUILD/notmuch-0.37/test --load test-lib.el --no-window-system --eval (setq server-name "notmuch-test-suite-15324") --eval (server-start) --eval (orphan-watchdog 15324)
mjg       694506  0.0  0.0   2868  1536 ?        S    17:37   0:00 emacsclient --socket-name=notmuch-test-suite-15324 --eval (notmuch-test-progn (let ((message-send-mail-function (lambda () t)) ?       (mail-host-address "example.com")) ?   (notmuch-mua-mail) ?   (message-goto-to) ?   (insert "test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000") ?   (message-goto-subject) ?   (insert "test encrypted message for cleartext index 001") ?   (message-goto-body) ?   (insert "This is a test encrypted message with a wumpus.\n") ?   (mml-secure-message-encrypt) ?   (let ((mml-secure-smime-sign-with-sender t) ?? (mml-secure-openpgp-sign-with-sender t)) ?     (notmuch-mua-send-and-exit))))
mjg       694516  0.0  0.0   8712  4608 ?        SLs  17:37   0:00 /usr/bin/gpg2 --no-tty --status-fd 1 --yes --command-fd 0 --armor --textmode --output /tmp/epg-output26aEdA --always-trust --encrypt -r 6D92612D94E46381

[-- Attachment #5: Type: text/plain, Size: 0 bytes --]

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:

> oh well, attachments ...
>

Can you encrypt to the key 6D92612D94E46381 interactively using an
approriately simplified version of that command?

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

David Bremner venit, vidit, dixit 2023-05-12 21:17:45:
> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
> 
> > oh well, attachments ...
> >
> 
> Can you encrypt to the key 6D92612D94E46381 interactively using an
> approriately simplified version of that command?

Took me a while, sorry. In that chroot:

```
<mock-chroot> sh-5.2# gpg --no-tty --import ./test/gnupg-secret-key.asc
gpg: directory '/builddir/.gnupg' created
gpg: /builddir/.gnupg/trustdb.gpg: trustdb created
gpg: key 6D92612D94E46381: public key "Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)" imported
gpg: key 6D92612D94E46381: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

<mock-chroot> sh-5.2# echo supersecret | gpg -ear 6D92612D94E46381
gpg: C44D36DEAD54AB16: There is no assurance this key belongs to the named user

sub  rsa1024/C44D36DEAD54AB16 2011-02-05 Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
 Primary key fingerprint: 5AEA B11F 5E33 DCE8 75DD  B75B 6D92 612D 94E4 6381
      Subkey fingerprint: 8987 5467 478A A81C EBD5  2E7E C44D 36DE AD54 AB16

It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.

Use this key anyway? (y/N)
```

Confirming that works, of course. Also, `gpg --always-trust -ear 6D92612D94E46381` works.

```
<mock-chroot> sh-5.2# printf '%s:6:\n' "$FINGERPRINT" | gpg --quiet --batch --no-tty --import-ownertrust
gpg: inserting ownertrust of 6
```
(like test-lib.sh does) and then encryption works - no questions asked.
So, all that works.

Are all gpg related tests emacs based? Either gpg or emacs is the red
herring here, or both ...

Unfortunately I have no clue about emacs/lisp and cannot dig further
there. I just know it's 100% reproducible (for f39 mock on f38, all fedoras
in copr, but not f38 mock on f38). Stomped.

Cheers
Michael

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

[now from the subscribed address, sorry]

David Bremner venit, vidit, dixit 2023-05-12 21:17:45:

> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>
> > oh well, attachments ...
> >
>
> Can you encrypt to the key 6D92612D94E46381 interactively using an
> approriately simplified version of that command?

Took me a while, sorry. In that chroot:

```
<mock-chroot> sh-5.2# gpg --no-tty --import ./test/gnupg-secret-key.asc
gpg: directory '/builddir/.gnupg' created
gpg: /builddir/.gnupg/trustdb.gpg: trustdb created
gpg: key 6D92612D94E46381: public key "Notmuch Test Suite
<test_suite@notmuchmail.org> (INSECURE!)" imported
gpg: key 6D92612D94E46381: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

<mock-chroot> sh-5.2# echo supersecret | gpg -ear 6D92612D94E46381
gpg: C44D36DEAD54AB16: There is no assurance this key belongs to the named user

sub  rsa1024/C44D36DEAD54AB16 2011-02-05 Notmuch Test Suite
<test_suite@notmuchmail.org> (INSECURE!)
 Primary key fingerprint: 5AEA B11F 5E33 DCE8 75DD  B75B 6D92 612D 94E4 6381
      Subkey fingerprint: 8987 5467 478A A81C EBD5  2E7E C44D 36DE AD54 AB16

It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.

Use this key anyway? (y/N)
```

Confirming that works, of course. Also, `gpg --always-trust -ear
6D92612D94E46381` works.

```
<mock-chroot> sh-5.2# printf '%s:6:\n' "$FINGERPRINT" | gpg --quiet
--batch --no-tty --import-ownertrust
gpg: inserting ownertrust of 6
```
(like test-lib.sh does) and then encryption works - no questions asked.
So, all that works.

Are all gpg related tests emacs based? Either gpg or emacs is the red
herring here, or both ...

Unfortunately I have no clue about emacs/lisp and cannot dig further
there. I just know it's 100% reproducible (for f39 mock on f38, all fedoras
in copr, but not f38 mock on f38). Stomped.

Cheers
Michael

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

Michael J Gruber <gruber@math.uni-hannover.de> writes:

> David Bremner venit, vidit, dixit 2023-05-12 21:17:45:
>> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>> 
>> > oh well, attachments ...
>> >
>> 
>> Can you encrypt to the key 6D92612D94E46381 interactively using an
>> approriately simplified version of that command?
>
> Took me a while, sorry. In that chroot:
>
> ```
> <mock-chroot> sh-5.2# gpg --no-tty --import ./test/gnupg-secret-key.asc
> gpg: directory '/builddir/.gnupg' created
> gpg: /builddir/.gnupg/trustdb.gpg: trustdb created
> gpg: key 6D92612D94E46381: public key "Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)" imported
> gpg: key 6D92612D94E46381: secret key imported
> gpg: Total number processed: 1
> gpg:               imported: 1
> gpg:       secret keys read: 1
> gpg:   secret keys imported: 1
>
> <mock-chroot> sh-5.2# echo supersecret | gpg -ear 6D92612D94E46381
> gpg: C44D36DEAD54AB16: There is no assurance this key belongs to the named user
>
> sub  rsa1024/C44D36DEAD54AB16 2011-02-05 Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
>  Primary key fingerprint: 5AEA B11F 5E33 DCE8 75DD  B75B 6D92 612D 94E4 6381
>       Subkey fingerprint: 8987 5467 478A A81C EBD5  2E7E C44D 36DE AD54 AB16
>

I'm not sure if this is the main issue, but in commit bf8aa34324cc91a
that key was replaced by an ed255519 key
9A3AFE6C60065A148FD4B58A7E6ABE924645CC60

Did the fedora sources get out of synch somehow?

d

PS. Currently spinning up a Fedora Rawhide VM to see if I can duplicate
the issue without the packaging infrastructure.

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
>
> Are all gpg related tests emacs based? Either gpg or emacs is the red
> herring here, or both ...

The issue (at least on rawhide) seems to be the interaction between gpg
and emacs

    https://dev.gnupg.org/T6481

A fix seems to be in progress. Do you have problems with gpg versions
other than 2.4.1?

d

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

David Bremner <david@tethera.net> writes:

>
> I'm not sure if this is the main issue, but in commit bf8aa34324cc91a
> that key was replaced by an ed255519 key
> 9A3AFE6C60065A148FD4B58A7E6ABE924645CC60
>
> Did the fedora sources get out of synch somehow?
>
> d
>
> PS. Currently spinning up a Fedora Rawhide VM to see if I can duplicate
> the issue without the packaging infrastructure.

Sorry, that's a silly question, that change is only in master so far.

d

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

Am Sa., 27. Mai 2023 um 14:59 Uhr schrieb David Bremner <david@tethera.net>:
>
> David Bremner <david@tethera.net> writes:
>
> >
> > I'm not sure if this is the main issue, but in commit bf8aa34324cc91a
> > that key was replaced by an ed255519 key
> > 9A3AFE6C60065A148FD4B58A7E6ABE924645CC60
> >
> > Did the fedora sources get out of synch somehow?
> >
> > d
> >
> > PS. Currently spinning up a Fedora Rawhide VM to see if I can duplicate
> > the issue without the packaging infrastructure.
>
> Sorry, that's a silly question, that change is only in master so far.
>

Not silly at all.

I'm seeing this with release (my local mock tests; Fedora koji) and
master (Fedora copr). Both build an rpm from a complete tarball (as
released resp. git-archive [via rpkg]), so the test suite is in-sync.

Cheers,
Michael

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

Am Sa., 27. Mai 2023 um 14:31 Uhr schrieb David Bremner <david@tethera.net>:
>
> Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
> >
> > Are all gpg related tests emacs based? Either gpg or emacs is the red
> > herring here, or both ...
>
> The issue (at least on rawhide) seems to be the interaction between gpg
> and emacs
>
>     https://dev.gnupg.org/T6481
>
> A fix seems to be in progress. Do you have problems with gpg versions
> other than 2.4.1?

Yes. "It's complicated". Packages in Fedora copr buildroots:

epel-8          gnupg2-2.2.20-3.el8_6   emacs-1:26.1-10.el8_8.2
epel-9          gnupg2-2.3.3-2.el9_0    emacs-27.2-8.el9_2.1
fedora-37       gnupg2-2.3.8-1.fc37     emacs-1:28.2-3.fc37
fedora-38       gnupg2-2.4.0-3.fc38     emacs-28.2-4.fc38
fedora-39       gnupg2-2.4.1-1.fc39     emacs-28.2-6.fc39
(39=rawhide)

RHEL/EPEL/ELN do not have "dtach" and thus skip the pertaining tests!
(I'll try this with dtach one day.)

notmuch master 0.37^62.gd155f29e (increased lisp timeout 120)
(workee epel-8 epel-9)
no workee fedora-37 fedora-38 fedora-39


notmuch 0.37 (standard lisp timeout 0.1)
no workee fedora-39
everything else works


notmuch 0.37  (increased lisp timeout 120)
(workee epel-8 epel-9)
no workee fedora-37 fedora-38 fedora-39

That is, on F37 and F38 failing:
T315-emacs-tagging
T460-emacs-tree

On F39 failing:
T315-emacs-tagging
T350-crypto
T357-index-decryption

Now, I don't know how the timeouts interact and whether increasing the
lisp timeout makes the overall timeout kick-in when it should not, of
course. Maybe this produces "false fails" on F37 and F38 because you
expect some lisp code to "hang" rather than return, and with the
increased timeout the test gets killed instead? "Fixing" T350/T357 for
the wrong reasons there, I guess. Based on that new hypothesis, I
omitted the lisp timeout again, and sure enough, it's only
fedora-39/rawhide failing again with T350/T357. So here is hoping that
gnupg2-2.4.0 vs gnupg2-2.4.1 indeed makes the difference!

Thanks,
Michael

Re: T350-crypto T357-index-decryption: possible race condition?

[Michael J Gruber]

Am Di., 30. Mai 2023 um 18:57 Uhr schrieb Michael J Gruber
<michaeljgruber+grubix+git@gmail.com>:
>
> Am Sa., 27. Mai 2023 um 14:31 Uhr schrieb David Bremner <david@tethera.net>:
> >
> > Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:
> > >
> > > Are all gpg related tests emacs based? Either gpg or emacs is the red
> > > herring here, or both ...
> >
> > The issue (at least on rawhide) seems to be the interaction between gpg
> > and emacs
> >
> >     https://dev.gnupg.org/T6481
> >

That was the perfect hint, it seems!

> On F39 failing:
> T315-emacs-tagging

This one I can fix by serializing the test suite. For whatever reasons
it fails on newer Fedoras when looping through the different tree mode
tests (in parallel).

> T350-crypto
> T357-index-decryption

And these I can fix by using gnupg2 2.4.2 plus the upstream fix from
the bug you pointed me to. Hooray!

This means that with gnupg2 2.4.1 and 2.4.2 the test suite may fail (I
guess emacs/gpg2 will be broken), unless your distro back-ports that
fix.
(I submitted it for Fedora.)

Cheers,
Michael

Re: T350-crypto T357-index-decryption: possible race condition?

[David Bremner]

Michael J Gruber <michaeljgruber+grubix+git@gmail.com> writes:

>
> This one I can fix by serializing the test suite. For whatever reasons
> it fails on newer Fedoras when looping through the different tree mode
> tests (in parallel).
>
>> T350-crypto
>> T357-index-decryption

Because parallel testing only affects running different T* scripts in
parallel, there must be some interaction between scripts (possibly just
load-related timing issues). 

I will wait for the gnupg2 fix to propagate to Fedora before I
investigate this. In the mean time, any hints to reproduce the failure
would be welcome.

d