unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
From: Gregor Zattler <telegraph@gmx.net>
To: notmuch <notmuch@notmuchmail.org>
Subject: consistent database corruption with notmuch new
Date: Sun, 13 Dec 2020 14:19:09 +0100	[thread overview]
Message-ID: <20201213131909.GD21521@no.workgroup> (raw)

Dear notmuch developers, I need help because notmuch
new on my configured notmuch system without
index/database consistently produces a corrupted
database.

On Friday my notmuch database got corrupted.
xapian-check xapian F did not fix the database.

I moved the xapian directory to xapian-corrupted and
did a notmuch new.  This consistently yields a
corrupted database after some hours (no other notmuch
running, no mail delivery, no cron jobs etc):

    1 (master *) grfz@no:~/Mail/.notmuch$ rm -rf xapian/
    0 (master *) grfz@no:~/Mail/.notmuch$ time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; zcat notmuch.dump.2020-12-11.1607641607 | notmuch tag --batch ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail-since-11 ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail.with-failure ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new
    Welcome to a new version of notmuch! Your database will now be upgraded.
    This process is safe to interrupt.
    Backing up tags to /home/grfz/Mail/.notmuch/dump-20201212T212701.gz...
    Your notmuch database has now been upgraded.
    Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
    Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
    Processed 1175458 total files in 2h 47m 16s (117 files/sec.).
    Added 1101746 new messages to the database.

    real    167m16,589s
    user    79m23,083s
    sys     6m19,064s
    Processed 124685 total files in 29m 21s (70 files/sec.).
    Added 83646 new messages to the database.

    real    29m25,353s
    user    10m24,737s
    sys     2m14,977s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Expected block 325393 to be level 0, not 1
    Processed 122555 total files in 30m 46s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m49,774s
    user    9m44,716s
    sys     2m3,859s
    adding tag frommeMessage-ID: (null)
    Status: A Xapian exception occurred
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 38s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m42,427s
    user    10m22,614s
    sys     2m2,737s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 31m 39s (64 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    31m43,529s
    user    10m33,959s
    sys     2m7,378s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 44s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m48,443s
    user    10m14,171s
    sys     2m8,209s


While the first notmuch new run seems OK, there at least
is the question why the second one adds another 83646
new messages.

This happened several times in a row.

This is a Thinkpad x240, 8GB RAM on debian buster but with
notmuch compiled from source (0.31.2+28~gadfded9).

I think the SSD is OK (no hints to SSD failures in log
files, smart tests OK, I even filled the free rest of
the SSD with randomm data and after that compared it
to the source with no difference.


Any idea what to do in order to get a running notmuch
mail system going?  If it's interesting to nomuch
development, what happens here, I'm happy to so some
digging if someone gives me instructions.  I'm not a
developer and have no clue how to debug this.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

             reply	other threads:[~2020-12-13 13:24 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-13 13:19 Gregor Zattler [this message]
2020-12-13 14:12 ` consistent database corruption with notmuch new David Bremner
2020-12-13 14:15   ` Gregor Zattler
2020-12-13 15:13     ` Gregor Zattler
2020-12-13 18:10       ` David Bremner
2020-12-13 18:12         ` David Bremner
2020-12-14 19:19           ` David Bremner
2020-12-13 21:22       ` Gregor Zattler
2020-12-14 19:22         ` Gregor Zattler
2021-01-30  8:54           ` out of memory on idle machine (was: Re: consistent database corruption with notmuch new) Gregor Zattler
2021-01-30 12:58             ` David Bremner
2021-01-31  8:16               ` out of memory on idle machine Gregor Zattler
2021-01-31 20:21                 ` Gregor Zattler
2021-02-03 11:32                   ` David Bremner
2021-02-03 11:59                 ` David Bremner
2021-02-07 21:46                   ` Gregor Zattler
2021-02-11 10:53                     ` David Bremner
2021-02-11 11:32                       ` David Bremner
2021-03-17 19:47                         ` bug: chokes on long directory names (was: Re: out of memory on idle machine) Gregor Zattler
2021-03-18  1:25                           ` [PATCH] test: add known broken test for long directory bug David Bremner
2021-03-18  7:26                             ` Tomi Ollila
2021-03-18 11:02                               ` David Bremner
2021-03-20 13:10                             ` [PATCH] lib/n_d_index_file: check return value from _n_m_add_filename David Bremner
2021-04-18 13:05                               ` David Bremner
2021-03-18  1:39                           ` bug: chokes on long directory names (was: Re: out of memory on idle machine) David Bremner
2021-02-12  4:19                       ` out of memory on idle machine Olly Betts
2021-02-21  9:42                         ` Gregor Zattler
2021-02-09  4:34                   ` Olly Betts
2021-02-13 20:30                     ` Gregor Zattler
2020-12-14  9:11 ` consistent database corruption with notmuch new David Edmondson
2020-12-14 12:27   ` Gregor Zattler

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://notmuchmail.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201213131909.GD21521@no.workgroup \
    --to=telegraph@gmx.net \
    --cc=notmuch@notmuchmail.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).