Re: Bug#842291: notmuch processes frequently stuck in select()

Re: Bug#842291: notmuch processes frequently stuck in select()

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

David Bremner <david@tethera.net> writes:

> Brian May <bam@debian.org> writes:
>> strace shows notmuch looping in select.
>>
>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>> etc
>>
>
> a backtrace would be helpful.
>
> d

Nevermind, I managed to download the list archive for debian-devel and
replicate the bug.

The bug seems to be related to smime signature verification. After
adding the attached mail message (and running notmuch-new), to replicate
the hang it suffices to run

% notmuch show --decrypt id:20161116T143809.GA.21721.stse@fsing.rootsland.net  

As far as workarounds, turning off decryption / signature verification
should allow you to at least view the thread.


[-- Attachment #2: 20161116T143809.GA.21721.stse@fsing.rootsland.net.eml:2, --]
[-- Type: application/octet-stream, Size: 10816 bytes --]

Return-path: <list@bendel.debian.org>
Received: from bendel.debian.org ([82.195.75.100])
	from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=clientcerts/bendel.debian.org,EMAIL=hostmaster@clientcerts/bendel.debian.org (verified)
	by master.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.84_2)
	(envelope-from <list@bendel.debian.org>)
	id 1c70fy-0000O8-Sk
	for debian-list-archive@master.debian.org; Wed, 16 Nov 2016 13:54:58 +0000
Received: by bendel.debian.org (Postfix, from userid 38)
	id C8DD72EB; Wed, 16 Nov 2016 13:54:58 +0000 (UTC)
Old-Return-Path: <stse@fsing.rootsland.net>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on bendel.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-8.1 required=4.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,DKIM_VERIFIED,LDO_WHITELIST,MURPHY_DRUGS_REL8,RP_MATCHES_RCVD
	autolearn=unavailable autolearn_force=no version=3.4.0
X-Original-To: lists-debian-devel@bendel.debian.org
Delivered-To: lists-debian-devel@bendel.debian.org
Received: from localhost (localhost [127.0.0.1])
	by bendel.debian.org (Postfix) with ESMTP id E8819297
	for <lists-debian-devel@bendel.debian.org>; Wed, 16 Nov 2016 13:54:50 +0000 (UTC)
X-Virus-Scanned: at lists.debian.org with policy bank en-ht
X-Amavis-Spam-Status: No, score=-7.391 tagged_above=-10000 required=5.3
	tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
	DKIM_VALID_AU=-0.1, LDO_WHITELIST=-5, MURPHY_DRUGS_REL8=0.02,
	RP_MATCHES_RCVD=-0.311] autolearn=ham autolearn_force=no
Received: from bendel.debian.org ([127.0.0.1])
	by localhost (lists.debian.org [127.0.0.1]) (amavisd-new, port 2525)
	with ESMTP id g6Yh2oZPVGq6 for <lists-debian-devel@bendel.debian.org>;
	Wed, 16 Nov 2016 13:54:46 +0000 (UTC)
X-policyd-weight: using cached result; rate: -6.1
X-Greylist: delayed 445 seconds by postgrey-1.35 at bendel; Wed, 16 Nov 2016 13:54:46 UTC
Received: from fsing.rootsland.net (fsing.rootsland.net [IPv6:2a01:4f8:c17:3852::2])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client did not present a certificate)
	by bendel.debian.org (Postfix) with ESMTPS id 3B18D174
	for <debian-devel@lists.debian.org>; Wed, 16 Nov 2016 13:54:45 +0000 (UTC)
Received: from osgiliath.gondor (osgiliath.gondor [IPv6:2a01:198:492::10])
	(using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "Stephan Seitz", Issuer "CAcert Class 3 Root" (not verified))
	by fsing.rootsland.net (Postfix) with ESMTPS id 3tJlt027RLz3wpF
	for <debian-devel@lists.debian.org>; Wed, 16 Nov 2016 14:46:48 +0100 (CET)
DKIM-Filter: OpenDKIM Filter v2.9.2 fsing.rootsland.net 3tJlt027RLz3wpF
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fsing.rootsland.net;
	s=mail20100714; t=1479304008;
	bh=oSsWUURYeCIiWKrX7yjTKTnOze7rfLLhNtsDNi7iMSA=;
	h=Date:From:To:Subject:References:In-Reply-To;
	b=eNAdDn42oVAl6/9eOsSLzr/VrP3MmhMZtKKDL+On1EE7kLPfLau+Am2Y0BLXhsGXn
	 uT80tgafwBG0nHs4XNL2pGsJNTXMW8ljshiFB1klzVQzS6ADG/cg6Tbj8Zr91qbL+S
	 FVEde7SH70yQ8z9c2E3P1LVTtzs0xVg8Cy812A9w=
Received: by osgiliath.gondor (Postfix, from userid 10009)
	id 3tJlsz5QGNzyLq; Wed, 16 Nov 2016 14:46:47 +0100 (CET)
Date: Wed, 16 Nov 2016 14:46:47 +0100
From: Stephan Seitz <stse+debian@fsing.rootsland.net>
To: debian-devel@lists.debian.org
Subject: Re: OpenSSL 1.1.0
Message-ID: <20161116T143809.GA.21721.stse@fsing.rootsland.net>
Mail-Followup-To: debian-devel@lists.debian.org
References: <20160611123036.GA8321@roeckx.be>
 <22571.12184.204754.473038@chiark.greenend.org.uk>
 <22572.20623.82302.920412@chiark.greenend.org.uk>
 <CAKcBokvqZwsdieqZg_5i8EGMZoGaXzeohmrx2P-pbzLSJ7LVNQ@mail.gmail.com>
 <20161116133128.gf6o7pueequkrwi5@bongo.bofh.it>
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=sha-256; boundary="axsomkoxwo2ilny4"
Content-Disposition: inline
In-Reply-To: <20161116133128.gf6o7pueequkrwi5@bongo.bofh.it>
Organization: Minas Tirith, Gondor
User-Agent: NeoMutt/20161104 (1.7.1)
X-Rc-Virus: 2007-09-13_01
X-Rc-Spam: 2008-11-04_01
Resent-Message-ID: <ykb-u-niu9M.A.79G.yUGLYB@bendel>
Resent-From: debian-devel@lists.debian.org
X-Mailing-List: <debian-devel@lists.debian.org> archive/latest/322800
X-Loop: debian-devel@lists.debian.org
List-Id: <debian-devel.lists.debian.org>
List-URL: <https://lists.debian.org/debian-devel/>
List-Post: <mailto:debian-devel@lists.debian.org>
List-Help: <mailto:debian-devel-request@lists.debian.org?subject=help>
List-Subscribe: <mailto:debian-devel-request@lists.debian.org?subject=subscribe>
List-Unsubscribe: <mailto:debian-devel-request@lists.debian.org?subject=unsubscribe>
Precedence: list
Resent-Sender: debian-devel-request@lists.debian.org
Resent-Date: Wed, 16 Nov 2016 13:54:58 +0000 (UTC)
Delivered-To: debian-list-archive@master.debian.org


--axsomkoxwo2ilny4
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mi, Nov 16, 2016 at 02:31:28 +0100, Marco d'Itri wrote:
>ChaCha20 is hardly obscure: if it is to you then I fear that your
>opinion on this issue is not informed enough to be useful.

It doesn=E2=80=99t matter in the end.

If no one wants to delay the next release until all applications support=20
OpenSSL 1.1.0 in a secure way even is upstream may not be interested in=20
patching the software for now then we can=E2=80=99t have version 1.1.0.

And there is still the problem that 1.1.0 is not supported as long as the=
=20
available LTS version.

Many greetings,

	Stephan

--=20
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |

--axsomkoxwo2ilny4
Content-Type: application/x-pkcs7-signature
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIIOkwYJKoZIhvcNAQcCoIIOhDCCDoACAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0B
BwGgggvqMIIGCDCCA/CgAwIBAgIBATANBgkqhkiG9w0BAQQFADB5MRAwDgYDVQQKEwdSb290
IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQg
U2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAe
Fw0wNTEwMTQwNzM2NTVaFw0zMzAzMjgwNzM2NTVaMFQxFDASBgNVBAoTC0NBY2VydCBJbmMu
MR4wHAYDVQQLExVodHRwOi8vd3d3LkNBY2VydC5vcmcxHDAaBgNVBAMTE0NBY2VydCBDbGFz
cyAzIFJvb3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCrSTURSHzSJn5TlM9D
qd0o10Iqi/OHeBlYfA+e2ol94fvrcpANdKGWZKufoCSZc9riVXbHF3v1BKxGuMO+f2SNEGwk
82GcwPKQ+lHm9WkBY8MPVuJKQs/iRIwlKKjFeQl9RrmK8+nzNCkIReQcn8uUBByBqBSzmGXE
Q+xOgo0J0b2qW42S0OzekMV/CsLj6+YxWl50PpczWejDAz1gM7/30W9HxM3uYoNSbi4ImqTZ
FRiRpoWSR7CuSOtttyHshRpocjWr//AQXcD0lKdq1TuSfkyQBX6TwSyLpI5idBVxbgtxA+qv
FTia1NIFcm+M+SvrWnIl+TlG43IbPgTDZCciECqKT1inA62+tC4T7V2qSNfVfdQqe1z6RgRQ
5MwOQluM7dvyz/yWk+DbETZUYjQ4jwxgmzuXVjit89Jbi6Bb6k6WuHzX1aCGcEDTkSm3ojyt
9Yy7zxqSiuQ0e8DYbF/pCsLDpyCaWt8sXVJcukfVm+8kKHA4IC/VfynAskEDaJLM4JzMl0tF
7zoQCqtwOpiVcK01seqFK6QcgCExqa5geoAmSAC4AcCTY1UikTxW56/bOiXzjzFU6iaLgVn5
odFTEcV7nQP2dBHgbbEsPyyGkZlxmqZ3izRg0RS0LKydr4wQ05/EavhvE/xzWfdmQnQeiuP4
3NJvmJzLR5iVQAX76QIDAQABo4G/MIG8MA8GA1UdEwEB/wQFMAMBAf8wXQYIKwYBBQUHAQEE
UTBPMCMGCCsGAQUFBzABhhdodHRwOi8vb2NzcC5DQWNlcnQub3JnLzAoBggrBgEFBQcwAoYc
aHR0cDovL3d3dy5DQWNlcnQub3JnL2NhLmNydDBKBgNVHSAEQzBBMD8GCCsGAQQBgZBKMDMw
MQYIKwYBBQUHAgEWJWh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZy9pbmRleC5waHA/aWQ9MTAwDQYJ
KoZIhvcNAQEEBQADggIBAH8IiKHaGlBJ2on7oQhy84r3HsQ6tHlbIDCxRd7CXdNlafHCXVRU
PIVfuXtCkcKZ/RtRm6tGpaEQU55tiKxzbiwzpvD0nuB1wT6IRanhZkP+VlrRekF490DaSjrx
C1uluxYG5sLnk7mFTZdPsR44Q4Dvmw2M77inYACHV30eRBzLI++bPJmdr7UpHEV5FpZNJ23x
HGzDwlVks7wU4vOkHx4y/CcVBc/dLq4+gmF78CEQGPZE6lM5+dzQmiDgxrvgu1pPxJnIB721
vaLbLmINQjRBvP+LivVRIqqIMADisNS8vmW61QNXeZvo3MhN+FDtkaVSKKKs+zZYPumUK5FQ
hxvWXtaMzPcPEAxSTtAWYeXlCmy/F8dyRlecmPVsYGN6b165Ti/Iubm7aoW8mA3t+T6XhDSU
rgCvoeXnkm5OvfPi2RSLXNLrAWygF6UtEOucekq9ve7O/e0iQKtwOIj1CodqwqsFYMlIBdpT
wd5Ed2qz8zw87YC8pjhKKSRf/lk7myV6VmMAZLldpGJ9VzZPrYPvH5JToI53V93lYRE9IwCQ
TDz6o2CTBKOvNfYOao9PSmCnhQVsRqGP9Md246FZV/dxssRuFFxtbUFm3xuTsdQAw+7Lzzw9
IYCpX2Nl/N3gX6T0K/CFcUHUZyX7GrGXrtaZghNB0m6lG5kngOcLqagAMIIF2jCCA8KgAwIB
AgIDApCCMA0GCSqGSIb3DQEBDQUAMFQxFDASBgNVBAoTC0NBY2VydCBJbmMuMR4wHAYDVQQL
ExVodHRwOi8vd3d3LkNBY2VydC5vcmcxHDAaBgNVBAMTE0NBY2VydCBDbGFzcyAzIFJvb3Qw
HhcNMTYwNDI1MTgzMTUxWhcNMTgwNDI1MTgzMTUxWjCBizEWMBQGA1UEAxMNU3RlcGhhbiBT
ZWl0ejEnMCUGCSqGSIb3DQEJARYYc3RzZUBmc2luZy5yb290c2xhbmQubmV0MSYwJAYJKoZI
hvcNAQkBFhdzdHNlQGZzaW5nLnJvb3RzbGFuZC5kZTEgMB4GCSqGSIb3DQEJARYRbnVyLWFi
LXNhbEBnbXguZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsChT1bTyg2ybv
SUF7f1GXu+O7A+8iGqIWtH0oljh6KHMYJDoy67JXl5sz2gFtPdwkRCpIrGbt/pHrYMda73cI
tLq3ZGuCR5gQ1Y6SV4sODCSfq8DPtY6pPhZtFocuMRMPSd1OID9p647dl7yk1aAZiT3WcK9T
CWQ5J+qF/SurGo6ccSDXV4XmmmAdpVaJ0qAq26E0GNUoxySR+CvtJ2q1H3UV4VzyqeDXbQi7
U87lZTgDQWPvgmHpiT2PcRyhC0xsQ7g3yw4UdtvF9SWBKO6BINmdtfYphTIRYAOxVRDi5DyG
Eziji5RjGH/tp/Fh8lFc4jB5fl4rhEUrGF0GSDvlAgMBAAGjggF7MIIBdzAMBgNVHRMBAf8E
AjAAMFYGCWCGSAGG+EIBDQRJFkdUbyBnZXQgeW91ciBvd24gY2VydGlmaWNhdGUgZm9yIEZS
RUUgaGVhZCBvdmVyIHRvIGh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZzAOBgNVHQ8BAf8EBAMCA6gw
QAYDVR0lBDkwNwYIKwYBBQUHAwQGCCsGAQUFBwMCBgorBgEEAYI3CgMEBgorBgEEAYI3CgMD
BglghkgBhvhCBAEwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzABhhZodHRwOi8vb2NzcC5j
YWNlcnQub3JnMDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuY2FjZXJ0Lm9yZy9jbGFz
czMtcmV2b2tlLmNybDBPBgNVHREESDBGgRhzdHNlQGZzaW5nLnJvb3RzbGFuZC5uZXSBF3N0
c2VAZnNpbmcucm9vdHNsYW5kLmRlgRFudXItYWItc2FsQGdteC5kZTANBgkqhkiG9w0BAQ0F
AAOCAgEAFLbVpkUUJU8WlCZHZkfX5rPhqCeVXS8WLVKqC4ZIPel7tqqpR5JNe5N5v8ZOv6Oi
uEY8gXvmkWUoRSgTZO9U3wgjFXHoUprWBclW7ATC5M8Eaa2eRqq819oS42V5fmdu+J9aVqQu
RVABIFrZeCLC7/LcESec0EfucN9tPMqPIyp9zLiIAlB/rElcoAc1Fo63Wvb6k8AHSrFYoyQP
tBnsNtN/KrlkXG0xq149YYCeDyLQZb9cX+HNI9Lkl4X5Pwy0gYjliIBr4NUzqNkpa1JgO6SB
VtrMj8lYDhK+2v/Y3U3b+Ko3ZCXJbwslGrdG5Pdmx4EMGyNPy8akY5KHTQy6ztmQGHVS9WPT
1QlhJ6x5IHqemzPhyjlHlSVUjzy8mbn8DU31AsOFL8RdSm2+CqT3rhg5JuTj5QgWH1ggJa8r
oOINuwbHqLhM/1M2V8AkZYeuBj02jvKYZrxTpQ8Dnq/Jg4Q7j8Z+pfj87SthNcfnPOOoZdB5
N2aOeAOg2ODBb5Vadr3DAEr0gs1UlBdzzxSHWkmOC4nx7qPRoSpjXISyWNjBabeRrVL7l5MR
O5B/1CiuSimsPpzkLBwiZpjoIifhXmwRazY2PGyP+9RLUS/bYW7CCgTfvel77Lk6+3dSc2wt
e22+RWwl+2//ViECS+ktnhPO/MpGN6wK5RD8sU8e8L8xggJtMIICaQIBATBbMFQxFDASBgNV
BAoTC0NBY2VydCBJbmMuMR4wHAYDVQQLExVodHRwOi8vd3d3LkNBY2VydC5vcmcxHDAaBgNV
BAMTE0NBY2VydCBDbGFzcyAzIFJvb3QCAwKQgjANBglghkgBZQMEAgEFAKCB5DAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjExMTYxMzQ2NDdaMC8GCSqG
SIb3DQEJBDEiBCAppHQX2oAd2XSmFSDw6EWNc7QqZu0YgtKuw9sFrfb2dTB5BgkqhkiG9w0B
CQ8xbDBqMAsGCWCGSAFlAwQBKjALBglghkgBZQMEARYwCwYJYIZIAWUDBAECMAoGCCqGSIb3
DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0D
AgIBKDANBgkqhkiG9w0BAQEFAASCAQBzudRPmKhfxljRKpWq1Uhav8bpEWlft56TQr5AAsRi
HkQnVtQ9JDHp7ObJwVGg23Wm7GaVhl8sVnP31CuokbVsylt7lJYKebkvfLNyylAtbkiLC7UD
Szmj7pviMd3Cr0pOBpSAe1yQOzaPDdtWAGM8xA3AQzpjLhtv4Zqdwc5V8mf3E5aAqC1lc8aA
sV9XAhifOYJp/Gr1mi1dJ/lShLhGbBcPNmwn5HtauA4r4wzQdqa9NU4pXyohHljodeuGm7PM
AxYOYJup7LnLcRokin2WUNfpgxoUq69XqiQ6QD0dQ/qKor6mc1dF6talJ/7RVj+nVgcZa1uB
lnSBzM0G9hMy

--axsomkoxwo2ilny4--

Re: Bug#842291: notmuch processes frequently stuck in select()

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 4961 bytes --]

Control: affects 842291 + gpgsm dirmngr

On Wed 2016-11-23 03:50:40 -0500, David Bremner wrote:
> David Bremner <david@tethera.net> writes:
>
>> Brian May <bam@debian.org> writes:
>>> strace shows notmuch looping in select.
>>>
>>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>>> select(10, [9], [], NULL, {1, 0})       = 0 (Timeout)
>>> etc
>>>
>>
>> a backtrace would be helpful.
>>
>> d
>
> Nevermind, I managed to download the list archive for debian-devel and
> replicate the bug.
>
> The bug seems to be related to smime signature verification. After
> adding the attached mail message (and running notmuch-new), to replicate
> the hang it suffices to run
>
> % notmuch show --decrypt id:20161116T143809.GA.21721.stse@fsing.rootsland.net  
>
> As far as workarounds, turning off decryption / signature verification
> should allow you to at least view the thread.

I've noticed similar behavior, and it seems to correlate with gpgsm
asking dirmngr for an update to CRLs related to S/MIME certs.

some CRLs simply do not respond at all (resulting in a timeout), and
some do not respond, or are laggy, when accessed over tor specifically.

I see a couple possible ways to consider resolving this, none of them
great, and i don't know exactly how to implement any of them:

 0) turn off CRL updates entirely during s/mime signature verification

 1) do s/mime signature verification without CRL updates, but schedule
    CRL checks to happen in the background for dirmngr, so that future
    verifications will reflect the cert validity

 2) have dirmngr avoid checking CRLs that it knows it has already
    updated recently

 3) tell dirmngr to use much shorter CRL fetch timeouts


Some example traffic from my dirmngr that uses tor (complete with
timestamps indicating just how bad the delays can be):

Nov 22 14:08:24 alice dirmngr[11976]: no CRL available for issuer id 770B4DA5913F2572B9F679AE0819FB7D77572689
Nov 22 14:08:24 alice dirmngr[11976]: fetching CRL from 'http://crl.ll.mit.edu/getcrl/LLCA3'
Nov 22 14:08:44 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data
Nov 22 14:08:44 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host not found
Nov 22 14:08:44 alice dirmngr[11976]: error retrieving 'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host
Nov 22 14:08:44 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:08:45 alice dirmngr[11976]: no CRL available for issuer id 770B4DA5913F2572B9F679AE0819FB7D77572689
Nov 22 14:08:45 alice dirmngr[11976]: fetching CRL from 'http://crl.ll.mit.edu/getcrl/LLCA3'
Nov 22 14:09:05 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data
Nov 22 14:09:05 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host not found
Nov 22 14:09:05 alice dirmngr[11976]: error retrieving 'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host
Nov 22 14:09:05 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:09:05 alice dirmngr[11976]: no CRL available for issuer id 26FD002905277B015EE9B2A3C092A348F28A4C6B
Nov 22 14:09:05 alice dirmngr[11976]: fetching CRL from 'http://crl.startssl.com/sca-client1.crl'
Nov 22 14:09:25 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No data
Nov 22 14:09:25 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host not found
Nov 22 14:09:25 alice dirmngr[11976]: error retrieving 'http://crl.startssl.com/sca-client1.crl': Unknown host
Nov 22 14:09:25 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host
Nov 22 14:09:25 alice dirmngr[11976]: no CRL available for issuer id 26FD002905277B015EE9B2A3C092A348F28A4C6B
Nov 22 14:09:25 alice dirmngr[11976]: fetching CRL from 'http://crl.startssl.com/sca-client1.crl'
Nov 22 14:09:45 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No data
Nov 22 14:09:45 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host not found
Nov 22 14:09:45 alice dirmngr[11976]: error retrieving 'http://crl.startssl.com/sca-client1.crl': Unknown host
Nov 22 14:09:45 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host

that's a 20-second lag between each failed check, adding up to 80
seconds delay in rendering a single thread where 4 messages were signed
by S/MIME keys signed by two different authorities.

Fwiw, crl.ll.mit.edu doesn't seem to respond over tor on port 80 at all
in some cases, and in other cases takes nearly a minute to reply:

0 dkg@alice:/tmp/cdtemp.Ue45bu$ time wget -q 'http://crl.ll.mit.edu/getcrl/LLCA3'

real	0m0.694s
user	0m0.008s
sys	0m0.008s
0 dkg@alice:/tmp/cdtemp.Ue45bu$ time torsocks wget -q 'http://crl.ll.mit.edu/getcrl/LLCA3'

real	0m58.828s
user	0m0.008s
sys	0m0.008s
0 dkg@alice:/tmp/cdtemp.Ue45bu$ 


Any thoughts on the best way to pursue this?

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Bug#842291: notmuch processes frequently stuck in select()

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>  0) turn off CRL updates entirely during s/mime signature verification
>
>  1) do s/mime signature verification without CRL updates, but schedule
>     CRL checks to happen in the background for dirmngr, so that future
>     verifications will reflect the cert validity
>
>  2) have dirmngr avoid checking CRLs that it knows it has already
>     updated recently
>
>  3) tell dirmngr to use much shorter CRL fetch timeouts
>

>
> Any thoughts on the best way to pursue this?
>
>     --dkg

Maybe the issue is in gmime's usage of gpgme. If I understand correctly
(which is far from a sure thing), pkcs7_verify calls gpgme_op_verify
which is synchronous, and (apparently) does not support timeouts. An
alternate strategy would be to call gpgme_op_verify_start, and then call
gpgme_wait, which has a nonblocking mode. I don't really understand the
S/MIME model, but naively it seems OK for signature verification to fail
if the CRL check doesn't finish quickly.

d

Re: [pkg-gnupg-maint] Bug#842291: notmuch processes frequently stuck in select()

[Werner Koch]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

On Wed, 23 Nov 2016 18:19, dkg@fifthhorseman.net said:

>  0) turn off CRL updates entirely during s/mime signature verification

The gpgsm option is --disable-crl-checks.  

>  1) do s/mime signature verification without CRL updates, but schedule
>     CRL checks to happen in the background for dirmngr, so that future
>     verifications will reflect the cert validity

As above but use 

  dirmngr-client--url --load-crl URLOFCRL

You need to known the URL of the CRL, though.

>  2) have dirmngr avoid checking CRLs that it knows it has already
>     updated recently

A CRL carries a next-update date which is homored by dirmngr.  Further
dirmngr doesn't avoids to download a CRL unless 30 minutes have passed
since the lassed download.

>  3) tell dirmngr to use much shorter CRL fetch timeouts

gpgsm -k  --enable-crl-check --force-crl-refresh  USERID

> that's a 20-second lag between each failed check, adding up to 80

That seems to be caused by DNS lookups.  For example ADNS keeps on
trying even if it has received an ENETUNREACH and thus no UDP query
packet has been sent out.   We will very likely replace ADNS by a more
flexible library in the next GnuPG version.


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [pkg-gnupg-maint] Bug#842291: notmuch processes frequently stuck in select()

[David Bremner]

Werner Koch <wk@gnupg.org> writes:

> On Wed, 23 Nov 2016 18:19, dkg@fifthhorseman.net said:
>
>>  0) turn off CRL updates entirely during s/mime signature verification
>
> The gpgsm option is --disable-crl-checks.  
>
>>  1) do s/mime signature verification without CRL updates, but schedule
>>     CRL checks to happen in the background for dirmngr, so that future
>>     verifications will reflect the cert validity

A notmuch user reported on IRC that adding disable-crl-checks to
~/.gnupg/dirmngr.conf eliminated the long pauses when verifying s/mime
signatures.

This will prevent the user from noticing Certificate revokations, so
it's not without cost in security, but perhaps it's temporary workaround
until we figure out some better solution.

d