From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QBarAwkSU2D4XQAA0tVLHw (envelope-from ) for ; Thu, 18 Mar 2021 08:40:41 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6Ez0OggSU2CNNwAAbx9fmQ (envelope-from ) for ; Thu, 18 Mar 2021 08:40:40 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AE1102551B for ; Thu, 18 Mar 2021 09:40:40 +0100 (CET) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 088CC270EF; Thu, 18 Mar 2021 04:40:35 -0400 (EDT) X-Greylist: delayed 451 seconds by postgrey-1.36 at nmbug; Thu, 18 Mar 2021 04:40:31 EDT Received: from avasout02.plus.net (avasout02.plus.net [212.159.14.17]) by mail.notmuchmail.org (Postfix) with ESMTPS id C67A520613 for ; Thu, 18 Mar 2021 04:40:31 -0400 (EDT) Received: from bubblegen.co.uk ([80.229.236.194]) by smtp with ESMTP id Mo5OlhBAN7QW9Mo5PlwSXo; Thu, 18 Mar 2021 08:32:59 +0000 X-Clacks-Overhead: "GNU Terry Pratchett" X-CM-Score: 0.00 X-CNFS-Analysis: v=2.3 cv=H+BAP9Qi c=1 sm=1 tr=0 a=G4bc5lkgapKKm1P+Twxy3Q==:117 a=G4bc5lkgapKKm1P+Twxy3Q==:17 a=dESyimp9J3IA:10 a=QN0yt0NFAAAA:20 a=F4pkP8dKPhoqV7koh1MA:9 a=QEXdDO2ut3YA:10 a=QGmZVn2jeZHSLs_I:21 a=_W_S_7VecoQA:10 a=EFPFMrZjj9c9Y--pX09V:22 a=Z5ABNNGmrOfJ6cZ5bIyy:22 a=bWyr8ysk75zN3GCy5bjg:22 Received: from matthews-mbp.lan (unknown [192.168.86.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bubblegen.co.uk (Postfix) with ESMTPSA id E128216DCB for ; Thu, 18 Mar 2021 08:32:53 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bubblegen.co.uk; s=dkim; t=1616056374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=SLL/gHGI72YpkpNJNvaRPuK0sPEwCtIBp9QKuZRlaVU=; b=sKh72eshOW84VQQP9HghdzWZUre3xxmx3HK881sdjFei5idY4mycF1wpjUi48H67yDnzz4 2MP+6K3fU3TejljFXgblVkgjpzDyBkjif8wTS/ao1TaAMMuw42bkUl5J22si/bxx5Ql9nk gv5DCKg1XdDPK3K8aZ5BWEyxBo0TWOmgHNDR3uI7IojKzWpRnFasU3OErsa/HlCxqejFoa 8IxdXGmsXTAtEbe1dd6+c0I6GCo2BdcV6vpZaM5FIioHZAqTEbVwELtBiDZKyDEhOI1tyy ZmIRU89Pqa4TXwhp1xaHTl+NtvYv+Gc6WCYvFAHZV3HeeQ2iLzOjXpUW/umYjw== From: Matthew Lear Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Database locked when running hooks Message-Id: <9C1993DF-84BD-4199-A9C8-BADA98498812@bubblegen.co.uk> Date: Thu, 18 Mar 2021 08:32:53 +0000 To: notmuch@notmuchmail.org X-Mailer: Apple Mail (2.3654.60.0.2.21) X-CMAE-Envelope: MS4wfIQKUTv/ZPxs4FL5uPTKfIlbNMNjqA5BL8Mk8xBrmHwhKraZN9PZa4Yr4Di9NJqzSC0RFogYgf/vlmvWgIld/i/w5kyHNdTEWFTTG79Yi5+VeUk6oDNH JrT6Gyz7P8MSCPWzqnqDt1GOjJFEEypeCx7dKQEtJQ2JpPsZb4XngicgNDC8IIbDfJdd3GSpfPH/yg== Message-ID-Hash: XZEGEM2NDO2O5MOAQGA7RFU5JXKXKQDU X-Message-ID-Hash: XZEGEM2NDO2O5MOAQGA7RFU5JXKXKQDU X-MailFrom: matt@bubblegen.co.uk X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0 X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: multipart/mixed; boundary="===============9085342975357588811==" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616056840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:list-id: list-help:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=GlxPb6GF+dGI22mSvPLEJ3hzRNn/ERALGqZQ6DEio9o=; b=D9cOPjZpGCiZJ2BwV4SCQqmPHAl9nZmLmGfUIhq3CNPpN4AnlYYOwVD921dyslfbia+aX8 NWryF6l0zShwF7kSUgMsMYa5BpC2YhirWDAVxgAEJllpPk217YInFaLzrT2b7mT1IUx+T+ WBMCAkAUob9yU7QMeFxixL47FrGS16MltlqyRIB14C/6dKCrlew505Z4IAzd0OBtUgrn68 JTLH6A85kiIpijGUlF+124B3VvkJdt7VCi5IAr3rvbe8MHC3MdxSgU5GFyeYzcv4QFkHhg fy5eACq/6DzEIi+++lQtCAaMc/l6qUdK4I/BlFlIOkawpIzGj3FNj1Ey7EiG8w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616056840; a=rsa-sha256; cv=none; b=PJ2X4s9HO9ox9O1WtZvVVGi6tR1prxidqr25/kphxBdBxXYqDQjdrMR4DSNGOUkoI69PGN oTeobS2t0O8oyG6vERTqAOIWWlRxsMjOh+Q/0TwcFdF2wDv0In1rV+PVy/lWwZEtFpJm4a eQYm4ZyNOZB4JMtny3pkeqYLkbtqNSNWQLMBwYr2eNtI/fVzbiPsFFMsybuYlOe4+nqYL1 jptMXfC0PlEaBgAr/tz5AwBC6Ee1A61vRpmP223a8nhjefkrKcw71Ik17fChaW8PvIFpTK JlRw2zoIKEAiZYEDKevLHCRNmrfTtY6KzDEHsuCgfxNx/QQABhd3hLIdFoZBtg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=bubblegen.co.uk header.s=dkim header.b=sKh72esh; dmarc=fail reason="SPF not aligned (relaxed)" header.from=bubblegen.co.uk (policy=reject); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: 6.00 X-Spam: Yes Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=bubblegen.co.uk header.s=dkim header.b=sKh72esh; dmarc=fail reason="SPF not aligned (relaxed)" header.from=bubblegen.co.uk (policy=reject); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: AE1102551B X-Spam-Score: 6.00 X-Migadu-Spam: Yes X-Migadu-Scanner: scn0.migadu.com X-TUID: ATppBfDJdyXW --===============9085342975357588811== Content-Type: multipart/alternative; boundary="Apple-Mail=_BC50CD0A-2C2F-4F05-93CA-DF0F354D2D73" --Apple-Mail=_BC50CD0A-2C2F-4F05-93CA-DF0F354D2D73 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello all, I recently updated to notmuch 0.31.4+137~g6967dcb and made some changes = to the way I sync my mail using lieer. I wanted to utilise the pre-new and post-new notmuch hooks. However, when running an operation to import mail in the context of the = notmuch pre-new hook (by running gmi pull), I can see that the database = is locked, which means that running notmuch new hangs. =46rom man (5) notmuch-hooks the wording is such that it implies that = the database is not locked when the pre-new and post-new hooks are run. If I strace notmuch new, I can see that the database is opened for = writing before the pre-new hook is run, and it is not closed before the = hook is run. This means that any operation which needs write access to database (e.g. = lieer=E2=80=99s gmi pull) cannot obtain the database lock, so it hangs. The problem can be reproduced by creating = $DATABASEDIR/.notmuch/hooks/pre-new and having it perform a notmuch CLI = command to write to the DB, such as apply a tag. I described the problem in https://github.com/gauteh/lieer/issues/195 = Assuming the database should be able to be opened for writing in the = context of the pre-new (and post-new) hooks, this seems like a bug. Here is some output... Running notmuch new before the pre-new hook is executed: .. openat(AT_FDCWD, = "/home/myhome/path/to/maildir//.notmuch/xapian/flintlock", = O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) =3D 3 fcntl(3, F_OFD_SETLKW, {l_type=3DF_WRLCK, l_whence=3DSEEK_SET, = l_start=3D0, l_len=3D1}) =3D 0 openat(AT_FDCWD, "/home/ml943601/b/mail/brcm//.notmuch/xapian/iamglass", = O_RDONLY) =3D 4 .. It=E2=80=99s never closed before the hook is executed. What the hang looks like: [~] $ notmuch new path: /home/myhome/path/to/maildir/gmaildir/ pull: partial synchronization.. (hid: 66446860) fetching changes ....fetching changes ....fetching changes ....fetching = changes ...resolving changes (376) = ........................................receiving content (156) ...^C Traceback (most recent call last): File "/home/myhome/git/lieer/gmi", line 24, in g.main () File "/home/myhome/git/lieer/lieer/gmailieer.py", line 215, in main args.func (args) File "/home/myhome/git/lieer/lieer/gmailieer.py", line 418, in pull self.partial_pull () File "/home/myhome/git/lieer/lieer/gmailieer.py", line 548, in = partial_pull updated =3D self.get_content (message_gids) File "/home/myhome/git/lieer/lieer/gmailieer.py", line 754, in = get_content self.remote.get_messages (need_content, _got_msgs, 'raw') File "/home/myhome/git/lieer/lieer/remote.py", line 133, in func_wrap return func (self, *args, **kwargs) File "/home/myhome/git/lieer/lieer/remote.py", line 384, in = get_messages cb (msg_batch) File "/home/myhome/git/lieer/lieer/gmailieer.py", line 749, in = _got_msgs with notmuch.Database (mode =3D notmuch.Database.MODE.READ_WRITE) as = db: File "/usr/local/lib/python3.8/dist-packages/notmuch/database.py", = line 164, in __init__ self.open(path, mode) File "/usr/local/lib/python3.8/dist-packages/notmuch/database.py", = line 223, in open status =3D Database._open(_str(path), mode, byref(db)) There is no other program running at the time of the hang which accesses = the database. The problem seems to be caused by the database being opened for writing = before the hook is executed. Does notmuch require the database to be open when calling the hook? Cheers, =E2=80=94 Matt --Apple-Mail=_BC50CD0A-2C2F-4F05-93CA-DF0F354D2D73 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello= all,
I recently updated to notmuch 0.31.4+137~g6967dcb = and made some changes to the way I sync my mail using lieer.
I wanted to utilise the pre-new and post-new notmuch = hooks.
However, when running an operation to import = mail in the context of the notmuch pre-new hook (by running gmi pull), I = can see that the database is locked, which means that running notmuch = new hangs.
=46rom man (5) notmuch-hooks the wording = is such that it implies that the database is not locked when the pre-new = and post-new hooks are run.

If I strace notmuch new, I can see that the database is = opened for writing before the pre-new hook is run, and it is not closed = before the hook is run.
This means that any = operation which needs write access to database (e.g. lieer=E2=80=99s gmi = pull) cannot obtain the database lock, so it hangs.

The problem can be = reproduced by creating $DATABASEDIR/.notmuch/hooks/pre-new and = having it perform a notmuch CLI command to write to the DB, such as = apply a tag.

I = described the problem in https://github.com/gauteh/lieer/issues/195

Assuming the database = should be able to be opened for writing in the context of the pre-new = (and post-new) hooks, this seems like a bug.
Here = is some output...

Running notmuch new before the pre-new hook is = executed:

..
openat(AT_FDCWD, = "/home/myhome/path/to/maildir//.notmuch/xapian/flintlock", = O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) =3D 3
fcntl(3, F_OFD_SETLKW, {l_type=3DF_WRLCK, l_whence=3DSEEK_SET, = l_start=3D0, l_len=3D1}) =3D 0
openat(AT_FDCWD, = "/home/ml943601/b/mail/brcm//.notmuch/xapian/iamglass", O_RDONLY) =3D = 4
..

It=E2=80=99s never closed before the = hook is executed.
What the hang looks = like:

 [~]
$ notmuch new
path: /home/myhome/path/to/maildir/gmaildir/
pull: partial synchronization.. (hid: 66446860)
fetching changes ....fetching changes ....fetching changes = ....fetching changes ...resolving changes (376) = ........................................receiving content (156) = ...^C
Traceback (most recent call last):
  File "/home/myhome/git/lieer/gmi", line 24, in = <module>
    g.main ()
  File "/home/myhome/git/lieer/lieer/gmailieer.py", line = 215, in main
    args.func = (args)
  File = "/home/myhome/git/lieer/lieer/gmailieer.py", line 418, in pull
    self.partial_pull ()
  = File "/home/myhome/git/lieer/lieer/gmailieer.py", line 548, in = partial_pull
    updated     =3D = self.get_content (message_gids)
  File = "/home/myhome/git/lieer/lieer/gmailieer.py", line 754, in = get_content
    self.remote.get_messages = (need_content, _got_msgs, 'raw')
  File = "/home/myhome/git/lieer/lieer/remote.py", line 133, in = func_wrap
    return func (self, *args, = **kwargs)
  File = "/home/myhome/git/lieer/lieer/remote.py", line 384, in = get_messages
    cb (msg_batch)
  File "/home/myhome/git/lieer/lieer/gmailieer.py", line = 749, in _got_msgs
    with = notmuch.Database (mode =3D notmuch.Database.MODE.READ_WRITE) as = db:
  File = "/usr/local/lib/python3.8/dist-packages/notmuch/database.py", line 164, = in __init__
    self.open(path, = mode)
  File = "/usr/local/lib/python3.8/dist-packages/notmuch/database.py", line 223, = in open
    status =3D = Database._open(_str(path), mode, byref(db))

There is no other program running at = the time of the hang which accesses the database.
The= problem seems to be caused by the database being opened for writing = before the hook is executed.

Does notmuch require the database to be = open when calling the hook?

Cheers,
=E2=80=94  Matt

= --Apple-Mail=_BC50CD0A-2C2F-4F05-93CA-DF0F354D2D73-- --===============9085342975357588811== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============9085342975357588811==--