How about this patch?

You'll need to set “notmuch-crypto-get-keys-asynchronously” to “t” to
see any benefit.