From: Gregor Zattler <telegraph@gmx.net>
To: notmuch <notmuch@notmuchmail.org>
Subject: Re: out of memory on idle machine
Date: Sun, 31 Jan 2021 21:21:39 +0100 [thread overview]
Message-ID: <20210131202139.GC5640@no.workgroup> (raw)
In-Reply-To: <20210131081638.GA5640@no.workgroup>
Hi David, notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [31. Jan. 2021]:
> I'll redo idea #2 on my laptop and will report it's results.
Now on the laptop:
grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210131T080546.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 1184742 total files in 3h 33m 35s (92 files/sec.).
Added 1115147 new messages to the database.
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%y" ~/Mail/inbox/cur
2021-01-31 12:14:39.771049424 +0100
0 (master *) grfz@no:~/Mail/.notmuch$
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
bash: quest: command not found
127 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian
Value 0 for record #1114128: 1.61208e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
So I think that's OK on my laptop, after the first notmuch
new. Now I do another one.
0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 121712 total files in 45m 4s (45 files/sec.).
Added 85345 new messages to the database. Removed 3 messages.
0 (master *) grfz@no:~/Mail/.notmuch$
The Problem remains, but at a different scale: In between
these two notmuch new runs there were only a few hours,
there's no way I received 121712 or 85345 emails in this time
frame.
0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612101986.719_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612105584.10821_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612109187.21153_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612112775.31326_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612116375.9385_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612120297.21133_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
terminate called after throwing an instance of 'Xapian::DatabaseCorruptError'
Aborted
134 (master *) grfz@no:~/Mail/.notmuch$
So here is the database corruption again, while doing the
third notmuch new.
I only now realize I did not do the tests after the second
notmuch new, but I copied the database so here is it:
0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian-2/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$
I don't have a clue on how to interpret this. Any ideas?
Thanks for your attention, Gregor
P.S.: These are the properties of the laptops root filesystem:
0 (master *) grfz@no:~$ sudo tune2fs -l /dev/mapper/vg-lv--root
[sudo] password for grfz:
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name: root
Last mounted on: /
Filesystem UUID: 801d2942-610d-4f43-9824-b275642e75de
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 14082048
Block count: 56308736
Reserved block count: 2815436
Free blocks: 23262312
Free inodes: 11957831
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stripe width: 8191
Flex block group size: 16
Filesystem created: Sat Nov 23 11:59:12 2019
Last mount time: Sun Jan 31 08:43:15 2021
Last write time: Sun Jan 31 08:43:00 2021
Mount count: 1
Maximum mount count: 23
Last checked: Sun Jan 31 08:43:00 2021
Check interval: 0 (<none>)
Lifetime writes: 7448 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
First orphan inode: 13902671
Default directory hash: half_md4
Directory Hash Seed: 22422f4f-fec4-4a9b-8ee6-5be99ecb21f4
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0x69629743
next prev parent reply other threads:[~2021-01-31 20:22 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-13 13:19 consistent database corruption with notmuch new Gregor Zattler
2020-12-13 14:12 ` David Bremner
2020-12-13 14:15 ` Gregor Zattler
2020-12-13 15:13 ` Gregor Zattler
2020-12-13 18:10 ` David Bremner
2020-12-13 18:12 ` David Bremner
2020-12-14 19:19 ` David Bremner
2020-12-13 21:22 ` Gregor Zattler
2020-12-14 19:22 ` Gregor Zattler
2021-01-30 8:54 ` out of memory on idle machine (was: Re: consistent database corruption with notmuch new) Gregor Zattler
2021-01-30 12:58 ` David Bremner
2021-01-31 8:16 ` out of memory on idle machine Gregor Zattler
2021-01-31 20:21 ` Gregor Zattler [this message]
2021-02-03 11:32 ` David Bremner
2021-02-03 11:59 ` David Bremner
2021-02-07 21:46 ` Gregor Zattler
2021-02-11 10:53 ` David Bremner
2021-02-11 11:32 ` David Bremner
2021-03-17 19:47 ` bug: chokes on long directory names (was: Re: out of memory on idle machine) Gregor Zattler
2021-03-18 1:25 ` [PATCH] test: add known broken test for long directory bug David Bremner
2021-03-18 7:26 ` Tomi Ollila
2021-03-18 11:02 ` David Bremner
2021-03-20 13:10 ` [PATCH] lib/n_d_index_file: check return value from _n_m_add_filename David Bremner
2021-04-18 13:05 ` David Bremner
2021-03-18 1:39 ` bug: chokes on long directory names (was: Re: out of memory on idle machine) David Bremner
2021-02-12 4:19 ` out of memory on idle machine Olly Betts
2021-02-21 9:42 ` Gregor Zattler
2021-02-09 4:34 ` Olly Betts
2021-02-13 20:30 ` Gregor Zattler
2020-12-14 9:11 ` consistent database corruption with notmuch new David Edmondson
2020-12-14 12:27 ` Gregor Zattler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210131202139.GC5640@no.workgroup \
--to=telegraph@gmx.net \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).