Bug: fatal error with notmuch new, second run starts indexing all over again

Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Dear notmuch developers,

notmuch new fails while indexing my emails and when executed a
second time it obviously starts over again.  This is on a
up-to-date debian stetch with notmuch from git:

0 (master) grfz@len:~/Mail$ notmuch new
Found 940740 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 659979 total files in 1h 43m 20s (106 files/sec.).
Added 584654 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master) grfz@len:~/Mail$ notmuch new
Error: A Xapian exception occurred. Halting processing.
Processed 660023 total files in 1h 44m 45s (105 files/sec.).
Added 584698 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master) grfz@len:~/Mail$ notmuch --version
notmuch 0.24.2+100~gab02265

The resutling xapian databases are not as large as I would
expect (an older xapian database from another config is 11 G
big): 

0 (master) grfz@len:~/Mail$ du -hs .
28G     .
0 (master) grfz@len:~/Mail$ du -hs .notmuch
7,1G    .notmuch

It's now not possible to search with notmuch:
0 (master) grfz@len:~/Mail$ notmuch search date:year..
0 (master) grfz@len:~/Mail$

Any ideas how to proceed in order to get a working notmuch setup?


Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

Gregor Zattler <telegraph@gmx.net> writes:

> Dear notmuch developers,
>
> notmuch new fails while indexing my emails and when executed a
> second time it obviously starts over again.  This is on a
> up-to-date debian stetch with notmuch from git:
> Any ideas how to proceed in order to get a working notmuch setup?
>

> Ciao, Gregor

Some things to try

1. Check disk space
2. run xapian-fsck
3. run "notmuch new" under gdb to try to get a backtrace
4. use "git bisect" to try to find what commit introduced this problem

d



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner <david@tethera.net> [03. Jul. 2017]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> notmuch new fails while indexing my emails and when executed a
>> second time it obviously starts over again.  This is on a
>> up-to-date debian stetch with notmuch from git:
>> Any ideas how to proceed in order to get a working notmuch setup?

> 1. Check disk space

Sorry, forgot to mention: there is enough free space on RAM (8
GB) and SSD (40 G free).

> 2. run xapian-fsck

xapian-check does not find any errors, only spelling and synonym
are "Lazily created, and not yet used."

> 3. run "notmuch new" under gdb to try to get a backtrace

I have no clue about gdb, there was no backtrace:

0 (master) grfz@len:~$ gdb --args notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from notmuch...done.
(gdb) run
Starting program: /usr/local/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Error: A Xapian exception occurred. Halting processing.
Processed 662357 total files in 1h 52m 51s (97 files/sec.).
Added 587023 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 4509) exited with code 01]
(gdb) bt
No stack.
(gdb) 


> 4. use "git bisect" to try to find what commit introduced this problem

I'll do so in the next few days.

Thanks, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> I have no clue about gdb, there was no backtrace:
>
...
> Error: A Xapian exception occurred. Halting processing.
> Processed 662357 total files in 1h 52m 51s (97 files/sec.).
> Added 587023 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 4509) exited with code 01]
> (gdb) bt
> No stack.
> (gdb) 

Try setting a break in _notmuch_database_log (b _notmuch_database_log)
and run "bt" at that break point. It might (or might not) be worth
continuing after the first breakpoint to inspect other errors.

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Tomi Ollila]

You could also try notmuch new --verbose 

--- and then look into the last file it was indexing when crash occurred...


Tomi

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

[-- Attachment #1: Type: text/plain, Size: 9682 bytes --]

Hi David, notmuch developers,
* David Bremner <david@tethera.net> [03. Jul. 2017]:
> Try setting a break in _notmuch_database_log (b _notmuch_database_log)
> and run "bt" at that break point. It might (or might not) be worth
> continuing after the first breakpoint to inspect other errors.

I did so with newest notmuch:

0 (master *) grfz@len:~/tmp$ notmuch --version
notmuch 0.24.2+84~g1ec6344
0 (master *) grfz@len:~/tmp$

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 941498 total files (that's not much mail).
q
Breakpoint 1, _notmuch_database_log (notmuch=0x555555825dd0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x555555825dd0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x555555825dd0, filename=filename@entry=0x55557b204400 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666048.R6065357166258349555.len:2,",
    message_ret=message_ret@entry=0x7fffffffd048) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x55557b204400 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666048.R6065357166258349555.len:2,", notmuch=0x555555825dd0) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x555574a15b80 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x55558520f1b0 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x0000555555567b44 in add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x5555638993b0 "/home/grfz/Mail/~ml", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#6  0x0000555555567b44 in add_files (notmuch=0x555555825dd0, path=path@entry=0x5555557cf710 "/home/grfz/Mail", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#7  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#8  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 660118 total files in 8h 59m 43s (20 files/sec.).
Added 584749 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 14593) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)



Since the gdb output shows a filename I looked at its contents:
There is a second "Content-Length:" -Header which does not make
sense but otherwise should not be a problem.  

Since its an email from an open mailing list I attach it to
this very email.

I then configured notmuch to scan the whole mail folder
/home/grfz/Mail/~ml/emacs-orgmode@gnu.org: 

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114424 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2780, filename=filename@entry=0x555558118ba0 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666090.R8081408151026683680.len:2,",
    message_ret=message_ret@entry=0x7fffffffd2c8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x555558118ba0 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666090.R8081408151026683680.len:2,", notmuch=0x5555557d2780) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2780, path=path@entry=0x5555557d25f0 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2780, path=path@entry=0x5555557d4f90 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 24m 51s (46 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 23253) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)


--> This time it's another file with another email but at closer
inspection it shows it also has two "Content-Length:" -headers.  For
the sake of consistency I attach the file to this very email.


I then created a test maildir folder with the first offending email as
the very only email and configured notmuch to only scan this test
maildir folder: *no problem*.  I the put the second offending email in
the test maildir folder and tried again: *no problem*



I then rescanned the maildir folder with emacs-org-mode mailing list
emails with notmuch from debian stretch:
0 (master=) grfz@len:~/src/notmuch$ rm -rf /home/grfz/Mail/~ml/emacs-orgmode@gnu.org/.notmuch/
0 (master=) grfz@len:~/src/notmuch$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/notmuch...(no debugging symbols
found)...done.
(gdb) b _notmuch_database_log
Function "_notmuch_database_log" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_notmuch_database_log) pending.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114424 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 13m 45s (84 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 24459) exited with code 01]
(gdb) bt
No stack.
(gdb) continue
The program is not being run.
(gdb)




Since notmuch from debian strech also produced a xapian exeption I
assume bisecting will be of no help.  I'll try to bisect but the next
week or so I don't have much free time.  I also will run memtest and
smartmontools.


Thanks for your help, Gregor

[-- Attachment #2: 1488666048.R6065357166258349555.len:2, --]
[-- Type: text/plain, Size: 5002 bytes --]

X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on shi.workgroup
X-Spam-grfz-Status: No, hits=-5.9 required=2.0 bayes=0.0000 tests=AWL=0.031,BAYES_00=-3.599,FREEMAIL_FROM=0.001,RCVD_IN_DNSWL_MED=-2.3,SPF_FAIL=0.001 autolearn=ham version=3.3.1 date="Tue, 19 Apr 2011 17:04:26 +0200" languages=
Envelope-to: grfz@localhost
Delivery-date: Tue, 19 Apr 2011 17:04:19 +0200
Received: from localhost ([127.0.0.1] helo=shi.workgroup)
	by shi.workgroup with esmtp (Exim 4.75)
	(envelope-from <error4me@gmx.de>)
	id 1QCC0T-0004yU-EE
	for grfz@localhost; Tue, 19 Apr 2011 16:34:21 +0200
Delivered-To: GMX delivery to telegraph@gmx.net
Received: from pop.gmx.net
	by shi.workgroup with POP3 (fetchmail-6.3.19)
	for <grfz@localhost> (single-drop); Tue, 19 Apr 2011 16:34:21 +0200 (CEST)
Received: (qmail invoked by alias); 10 Apr 2011 20:15:29 -0000
Received: from lists.gnu.org (EHLO lists.gnu.org) [199.232.76.165]
  by mx0.gmx.net (mx040) with SMTP; 10 Apr 2011 22:15:29 +0200
Received: from localhost ([127.0.0.1]:45444 helo=lists.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43)
	id 1Q912e-000695-PP
	for telegraph@gmx.net; Sun, 10 Apr 2011 16:15:28 -0400
Received: from [140.186.70.92] (port=42637 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q90w4-0000gL-Oq
	for emacs-orgmode@gnu.org; Sun, 10 Apr 2011 16:08:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <nick@dokosmarshall.org>) id 1Q90w3-0005GN-G7
	for emacs-orgmode@gnu.org; Sun, 10 Apr 2011 16:08:40 -0400
Received: from vms173003pub.verizon.net ([206.46.173.3]:45329)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <nick@dokosmarshall.org>) id 1Q90w3-0005D3-CJ
	for emacs-orgmode@gnu.org; Sun, 10 Apr 2011 16:08:39 -0400
Received: from alphaville.dokosmarshall.org ([unknown] [173.76.32.106])
	by vms173003.mailsrvcs.net
	(Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16
	2009)) with ESMTPA id <0LJG009M9D9DEKK0@vms173003.mailsrvcs.net> for
	emacs-orgmode@gnu.org; Sun, 10 Apr 2011 15:08:13 -0500 (CDT)
Received: by alphaville.dokosmarshall.org (Postfix, from userid 1000)
	id EDB32E25D1; Sun, 10 Apr 2011 16:08:00 -0400 (EDT)
Received: from alphaville.dokosmarshall.org (localhost [127.0.0.1])
	by alphaville.dokosmarshall.org (Postfix) with ESMTP id D8F1FE207E; Sun,
	10 Apr 2011 16:08:00 -0400 (EDT)
To: "Thomas S. Dye" <tsd@tsdye.com>
From: Nick Dokos <nicholas.dokos@hp.com>
Subject: Re: [O] LaTeX export: inline markup underscore problem
In-reply-to: Message from "Thomas S. Dye" <tsd@tsdye.com> of
	"Sun, 10 Apr 2011 09:36:35 -1000."
	<F9AF318F-73F7-428B-9BA0-2C3DF64F9428@tsdye.com>
References: <BANLkTimv45j42T2yOja2Ci2CQD-268V9uA@mail.gmail.com>
	<F9AF318F-73F7-428B-9BA0-2C3DF64F9428@tsdye.com>
X-Mailer: MH-E 8.2; nmh 1.3; GNU Emacs 24.0.50
Date: Sun, 10 Apr 2011 16:08:00 -0400
Message-id: <1112.1302466080@alphaville.dokosmarshall.org>
X-detected-operating-system: by eggs.gnu.org: Solaris 10 (1203?)
X-Received-From: 206.46.173.3
Cc: Mauro Rovezzi <mauro.rovezzi@gmail.com>, emacs-orgmode@gnu.org,
	nicholas.dokos@hp.com
X-BeenThere: emacs-orgmode@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: nicholas.dokos@hp.com
List-Id: "General discussions about Org-mode." <emacs-orgmode.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/emacs-orgmode>
List-Post: <mailto:emacs-orgmode@gnu.org>
List-Help: <mailto:emacs-orgmode-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=subscribe>
Sender: error4me@gmx.de
Errors-To: emacs-orgmode-bounces+telegraph=gmx.net@gnu.org
X-GMX-Antivirus: 0 (no virus found)
X-GMX-Antispam: 0 (Mail was not recognized as spam);
 Detail=5D7Q89H36p5c/wJqTWc2KfAmv6U3syj2DinrxktPnaNZhfrNCczkB8SdMXDFwycd4V9CQ
 w8px7DbZJcTWciL08KzEgtQhRFGEK2QJMUC6s/EvqM6i7d6APK/zw34uNDbk2rQSs8r9bt2pcPbO
 fTxmpZYFtp5aXxPyRJw2T1hCiuybvgjpHnHrtu1Rm8u2WdPdd/CZun9aTDYFYcHH+7oTw==V1;
Content-Length: 873

Content-Length: 852

Thomas S. Dye <tsd@tsdye.com> wrote:

> Aloha Mauro,
> 
> Please see
> 
> http://orgmode.org/worg/org-tutorials/org-latex-export.html#sec-17_2
> 

Excuse the intrusion, but I don't see anything in there that looks like
a solution to the OP's problem (the undesired quoting of the underscore
when exporting the org file to LaTeX): what am I missing?

I was going to suggest that life would be much easier for the OP if he
could change all underscores to some other character in his bibliography
(I often see colons and minus signs used e.g.) That may or may not be
possible of course, but if it is possible, it would be the simplest
solution: the treatment of _ in the LaTeX exporter is often the source
of frustration. That is not the fault of the exporter of course: TeX
makes it very complicated to determine what the right thing to do is.

Nick





[-- Attachment #3: 1488666090.R8081408151026683680.len:2, --]
[-- Type: text/plain, Size: 5318 bytes --]

X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on shi.workgroup
X-Spam-grfz-Status: No, hits=-5.2 required=2.0 bayes=0.0000 tests=AWL=-0.388,BAYES_00=-3.599,FREEMAIL_FROM=0.001,MISSING_HEADERS=1.021,RCVD_IN_DNSWL_MED=-2.3,SPF_FAIL=0.001,T_FRT_FOLLOW1=0.01,T_TO_NO_BRKTS_FREEMAIL=0.01 autolearn=ham version=3.3.1 date="Tue, 07 Jun 2011 06:01:00 +0200" languages=
Envelope-to: grfz@localhost
Delivery-date: Tue, 07 Jun 2011 05:59:17 +0200
Received: from localhost ([127.0.0.1] helo=shi.workgroup)
	by shi.workgroup with esmtp (Exim 4.76)
	(envelope-from <error4me@gmx.de>)
	id 1QTnRk-0006ZV-SS
	for grfz@localhost; Tue, 07 Jun 2011 05:59:17 +0200
Delivered-To: GMX delivery to telegraph@gmx.net
Received: from pop.gmx.net
	by shi.workgroup with POP3 (fetchmail-6.3.19)
	for <grfz@localhost> (single-drop); Tue, 07 Jun 2011 05:59:16 +0200 (CEST)
Received: (qmail invoked by alias); 06 Jun 2011 22:35:16 -0000
Received: from lists.gnu.org (EHLO lists.gnu.org) [140.186.70.17]
  by mx0.gmx.net (mx078) with SMTP; 07 Jun 2011 00:35:16 +0200
Received: from localhost ([::1]:43996 helo=lists.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <emacs-orgmode-bounces+telegraph=gmx.net@gnu.org>)
	id 1QTiOA-00065P-LE
	for telegraph@gmx.net; Mon, 06 Jun 2011 18:35:14 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41632)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <nicholas.dokos@hp.com>) id 1QTfys-0001f6-DO
	for emacs-orgmode@gnu.org; Mon, 06 Jun 2011 16:00:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <nicholas.dokos@hp.com>) id 1QTfyp-0003VB-S4
	for emacs-orgmode@gnu.org; Mon, 06 Jun 2011 16:00:57 -0400
Received: from g4t0014.houston.hp.com ([15.201.24.17]:12096)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <nicholas.dokos@hp.com>) id 1QTfyo-0003UI-Uu
	for emacs-orgmode@gnu.org; Mon, 06 Jun 2011 16:00:55 -0400
Received: from g4t0009.houston.hp.com (g4t0009.houston.hp.com [16.234.32.26])
	by g4t0014.houston.hp.com (Postfix) with ESMTP id 3E7F024207;
	Mon,  6 Jun 2011 20:00:53 +0000 (UTC)
Received: from alphaville.americas.hpqcorp.net
	(alphaville.americas.hpqcorp.net [16.117.225.242])
	by g4t0009.houston.hp.com (Postfix) with ESMTP id 0B7D5C1EA;
	Mon,  6 Jun 2011 20:00:53 +0000 (UTC)
Received: by alphaville.americas.hpqcorp.net (Postfix, from userid 1000)
	id B0037E28A6; Mon,  6 Jun 2011 16:00:52 -0400 (EDT)
Received: from alphaville.americas.hpqcorp.net (localhost [127.0.0.1])
	by alphaville.americas.hpqcorp.net (Postfix) with ESMTP id 9A733E2775; 
	Mon,  6 Jun 2011 16:00:52 -0400 (EDT)
From: Nick Dokos <nicholas.dokos@hp.com>
In-Reply-To: Message from Nick Dokos <nicholas.dokos@hp.com> of "Mon,
	06 Jun 2011 15:14:12 EDT."
	<4336.1307387652@alphaville.americas.hpqcorp.net>
References: <BANLkTimvGH01YfmZyMbzakYr1Dxdq_m=FQ@mail.gmail.com>
	<87hb82q3t7.fsf@gmail.com>
	<4336.1307387652@alphaville.americas.hpqcorp.net>
X-Mailer: MH-E 8.2; nmh 1.3; GNU Emacs 24.0.50
Date: Mon, 06 Jun 2011 16:00:52 -0400
Message-ID: <5057.1307390452@alphaville.americas.hpqcorp.net>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older,
	4)
X-Received-From: 15.201.24.17
Cc: Marvin Doyley <marvinpas@gmail.com>, nicholas.dokos@hp.com,
	emacs-orgmode@gnu.org
Subject: Re: [O] C-c - broken
X-BeenThere: emacs-orgmode@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
Reply-To: nicholas.dokos@hp.com
List-Id: "General discussions about Org-mode." <emacs-orgmode.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=unsubscribe>
List-Archive: </archive/html/emacs-orgmode>
List-Post: <mailto:emacs-orgmode@gnu.org>
List-Help: <mailto:emacs-orgmode-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/emacs-orgmode>,
	<mailto:emacs-orgmode-request@gnu.org?subject=subscribe>
Errors-To: emacs-orgmode-bounces+telegraph=gmx.net@gnu.org
Sender: error4me@gmx.de
X-GMX-Antivirus: 0 (no virus found)
X-GMX-Antispam: 0 (Sender is in whitelist: nicholas.dokos@hp.com);
 Detail=5D7Q89H36p4L00VTXC6D4q0N+AH0PUCnGxfMrHMdyAmW9ybpDu6/MhLNjkJhSju97FJFZ
 ovSReOomFAwSIuksg==V1;
Content-Length: 1096

Content-Length: 1074

Nick Dokos <nicholas.dokos@hp.com> wrote:

> Nicolas Goaziou <n.goaziou@gmail.com> wrote:
> 
> > Hello,
> > 
> > Marvin Doyley <marvinpas@gmail.com> writes:
> > 
> > > I use to use C-c - to convert headlines to plain list, but for some reason
> > > when I do this it tries to make a table of content, see below
> > >
> > > TABLE-OF-CONTENTS on ~/Documents/General/O/org/planner.org
> > > SPC=view TAB=goto RET=goto+hide [q]uit [r]escan [l]abels [f]ollow [x]r
> > > [?]Help
> > 
> > I cannot reproduce this. Could you be more explicit ? What is your test
> > file, where do you have cursor when you use C-c -, etc.
> > 
> 
> It looks as if C-c - has been remapped by some package. What does C-h k
> say for it? What minor modes are in effect in the planner.org buffer?
> Maybe if you ask for Help with ? in the context above, it'll give
> you an idea of who it is that's doing this.
> 

Looks like reftex is the culprit - you can blame the guy who created
reftex, what's his name? Ah, yes: Carsten Dominik :-)

But why do you have reftex turned on in an org buffer?

Nick

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> Since its an email from an open mailing list I attach it to
> this very email.
>

It might or might not be related, but I noticed that those two messages
get merged for me into a thread 1734 messages long of org-mode messages
(I have some old org messages, but I think I not the two you mention).

I'm not sure yet if it's a bug in notmuch, or something strange about
mail from the org-mode list.

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner <david@tethera.net> [09. Jul. 2017]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> Since its an email from an open mailing list I attach it to
>> this very email.
>>
> 
> It might or might not be related, but I noticed that those two messages
> get merged for me into a thread 1734 messages long of org-mode messages
> (I have some old org messages, but I think I not the two you mention).
> 
> I'm not sure yet if it's a bug in notmuch, or something strange about
> mail from the org-mode list.

This *might* be related with some other problem I reported
regarding false threading of messages, as discussed in the thread
containing amongst others id:874nvcekjk.fsf@qmul.ac.uk

Short version: Some of the messages on this mailinglist had very
weired References: headers mot probably causing notmuch to
misbehave while threading the messages.  But then there was no
xapian exeption involved.


Otherwise I did an extensive memory test: no prblem and an
smartctl -t long and -t offline test: no problems.  Remains to do
a bisecting but I have no free time atm.  Will report back later.

Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi David,
> * David Bremner <david@tethera.net> [09. Jul. 2017]:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> Since its an email from an open mailing list I attach it to
>>> this very email.
>>>
>> 
>> It might or might not be related, but I noticed that those two messages
>> get merged for me into a thread 1734 messages long of org-mode messages
>> (I have some old org messages, but I think I not the two you mention).
>> 
>> I'm not sure yet if it's a bug in notmuch, or something strange about
>> mail from the org-mode list.
>
> This *might* be related with some other problem I reported
> regarding false threading of messages, as discussed in the thread
> containing amongst others id:874nvcekjk.fsf@qmul.ac.uk
>
> Short version: Some of the messages on this mailinglist had very
> weired References: headers mot probably causing notmuch to
> misbehave while threading the messages.  But then there was no
> xapian exeption involved.
>

Right. I looks like there are indeed many message-ids in the resulting
database that look like valid email addresses. So that problem persists,
despite an attempted fix discussed in that thread.

I was wondering if the exception could result from overflowing some
internal limit due to threads many thousands of messages long. I will be
hard to know until I can replicate the exception. Perhaps a more
complete org-mode list archive would do it.

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner <david@tethera.net> [09. Jul. 2017]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> Short version: Some of the messages on this mailinglist had very
>> weired References: headers mot probably causing notmuch to
>> misbehave while threading the messages.  But then there was no
>> xapian exeption involved.
>>
> 
> Right. I looks like there are indeed many message-ids in the resulting
> database that look like valid email addresses. So that problem persists,
> despite an attempted fix discussed in that thread.
> 
> I was wondering if the exception could result from overflowing some
> internal limit due to threads many thousands of messages long. I will be
> hard to know until I can replicate the exception. Perhaps a more
> complete org-mode list archive would do it.

Quite possible, when I do 
notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/emacs-orgmode@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
and open the resulting mbox with mutt, it shows 10199 messages.

There is fun to have regarding message ids:

0 grfz@len:/tmp$ grep -c "^Message-ID:[[:space:]]" eom.mbox 
8758
0 grfz@len:/tmp$ grep -ci "^Message-ID:[[:space:]]" eom.mbox 
10174

I then split the mbox with mutt in a maildir folder.  This were
10199 individual files.  I then extracted the message ids via
formail from this files and piped this through sort -u.  These
were 10105 message ids.

The maildir with the emacs-orgmode@gnu.org emails contained 114563
individual files.  I removed every file somewhere containing one
of those message ids.  Now there were only 102164 individual files
in the maildir.

And after that I indexed this cleaned up maildir.  If somehow
this message threading problem affects the indexing it should now
index the files without xapian exeption:


This is with newest notmuch:
0 grfz@len:/tmp$ /home/grfz/src/notmuch/notmuch --version
notmuch 0.24.2+112~g37d1fa5


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 102164 total files (that's not much mail).
Processed 102164 total files in 5m 58s (284 files/sec.).
Added 102151 new messages to the database.
[Inferior 1 (process 26339) exited normally]
(gdb) 


While indexing the original emacs-org mode mailing list maildir
with it's 114563 files with the very same binary results in a
xapian exeption:


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114563 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2780, filename=filename@entry=0x555557ae2070 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666090.R8081408151026683680.len:2,",
    message_ret=message_ret@entry=0x7fffffffd2c8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x555557ae2070 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur/1488666090.R8081408151026683680.len:2,", notmuch=0x5555557d2780) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2780, path=path@entry=0x5555557d25f0 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2780, path=path@entry=0x5555557d4f90 "/home/grfz/Mail/~ml/emacs-orgmode@gnu.org", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 25m 59s (44 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 27880) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)

The referenced file is the second one I attached in this thread.




So your/Davids intuition was right.  The problem has to do with
this threading problem.


I also did extensive memtest and every kind of smartctl test
possible, to be sure this is no hardware problem.



I then downloaded parts of the archive, merged and sliced it and
now I have a sample of 25001 total files (that's not much mail),
on which notmuch new produces a xapian exeption:

0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 25001 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d27f0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d27f0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d27f0, filename=filename@entry=0x555558dc96c0 "/tmp/reduced-sample/cur/1499897912.R1843131171398763530.len:2,",
                                                                                              message_ret=message_ret@entry=0x7fffffffd2e8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555558dc96c0 "/tmp/reduced-sample/cur/1499897912.R1843131171398763530.len:2,", notmuch=0x5555557d27f0) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d27f0, path=path@entry=0x5555557e7950 "/tmp/reduced-sample/cur", state=state@entry=0x7fffffffd560) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d27f0, path=path@entry=0x5555557cf710 "/tmp/reduced-sample", state=state@entry=0x7fffffffd560) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda88) at notmuch.c:456
(gdb)


As a tar.xz it weighs 28 MB.  I could provide this for download
if someone is interested.


Ciao, Gregor

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi David, notmuch developers,
> * David Bremner <david@tethera.net> [09. Jul. 2017]:
>> Gregor Zattler <telegraph@gmx.net> writes:
>> 
>> I was wondering if the exception could result from overflowing some
>> internal limit due to threads many thousands of messages long. I will be
>> hard to know until I can replicate the exception. Perhaps a more
>> complete org-mode list archive would do it.
>
> Quite possible, when I do 
> notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/emacs-orgmode@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
> and open the resulting mbox with mutt, it shows 10199 messages.

Are those literal search terms "date 64 bit 32"?

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

Sure, send me a download URL.

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner <david@tethera.net> [13. Jul. 2017]:
>> * David Bremner <david@tethera.net> [09. Jul. 2017]:
>>> Gregor Zattler <telegraph@gmx.net> writes:
>>> I was wondering if the exception could result from overflowing some
>>> internal limit due to threads many thousands of messages long. I will be
>>> hard to know until I can replicate the exception. Perhaps a more
>>> complete org-mode list archive would do it.
>>
>> Quite possible, when I do 
>> notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/emacs-orgmode@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
>> and open the resulting mbox with mutt, it shows 10199 messages.
> 
> Are those literal search terms "date 64 bit 32"?

yes.

>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.

I'll do so in a private email.  


Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> I then downloaded parts of the archive, merged and sliced it and
> now I have a sample of 25001 total files (that's not much mail),
> on which notmuch new produces a xapian exeption:
>

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

For me this corpus does not produce an exception (although it does have
one thread with 5067 messages).

I'm running Debian testing, so there could be slightly different
versions of xapian and gmime. What versions of libxapian-dev and
libgmime-2.6-dev have you built notmuch against?

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner <david@tethera.net> [13. Jul. 2017]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> I then downloaded parts of the archive, merged and sliced it and
>> now I have a sample of 25001 total files (that's not much mail),
>> on which notmuch new produces a xapian exeption:
>>
> 
>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.
> 
> For me this corpus does not produce an exception (although it does have
> one thread with 5067 messages).

I cannot view this ATM because I cannot index these emails.  When
viewd with mutt there are no such gargantuan threads.

> I'm running Debian testing, so there could be slightly different
> versions of xapian and gmime. What versions of libxapian-dev and
> libgmime-2.6-dev have you built notmuch against?

Mhm.  I'm running debian stretch.  The versions are:

0 grfz@len:/tmp$ apt-cache policy libxapian-dev libgmime-2.6-dev
libxapian-dev:
Installed: 1.4.3-2
Candidate: 1.4.3-2
Version table:
*** 1.4.3-2 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
libgmime-2.6-dev:
Installed: 2.6.22+dfsg2-1
Candidate: 2.6.22+dfsg2-1
Version table:
*** 2.6.22+dfsg2-1 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status


BTW:


0 grfz@len:/tmp$ apt-cache policy gcc
gcc:
Installed: 4:6.3.0-4
Candidate: 4:6.3.0-4
Version table:
*** 4:6.3.0-4 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
0 grfz@len:/tmp$ gcc --version
gcc (Debian 6.3.0-18) 6.3.0 20170516



        

HTH, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi David,
> * David Bremner <david@tethera.net> [13. Jul. 2017]:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> I then downloaded parts of the archive, merged and sliced it and
>>> now I have a sample of 25001 total files (that's not much mail),
>>> on which notmuch new produces a xapian exeption:
>>>
>> 
>>> As a tar.xz it weighs 28 MB.  I could provide this for download
>>> if someone is interested.
>> 
>> For me this corpus does not produce an exception (although it does have
>> one thread with 5067 messages).
>
> I cannot view this ATM because I cannot index these emails.  When
> viewd with mutt there are no such gargantuan threads.
>
>> I'm running Debian testing, so there could be slightly different
>> versions of xapian and gmime. What versions of libxapian-dev and
>> libgmime-2.6-dev have you built notmuch against?
>
> Mhm.  I'm running debian stretch.  The versions are:
>

I tried on a debian stretch with the same library versions, and I still
can't duplicate the crash.

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner <david@tethera.net> [13. Jul. 2017]:
>> * David Bremner <david@tethera.net> [13. Jul. 2017]:
>>> Gregor Zattler <telegraph@gmx.net> writes:
>>>> I then downloaded parts of the archive, merged and sliced it and
>>>> now I have a sample of 25001 total files (that's not much mail),
>>>> on which notmuch new produces a xapian exeption:
[...]
>>> For me this corpus does not produce an exception (although it does have
>>> one thread with 5067 messages).

>>> I'm running Debian testing, so there could be slightly different
>>> versions of xapian and gmime. What versions of libxapian-dev and
>>> libgmime-2.6-dev have you built notmuch against?
[...]
> I tried on a debian stretch with the same library versions, and I still
> can't duplicate the crash.

This is strange.  I rerun the self compiled notmuch new on this
corpus and it did not throw an exeption, nore did the debian
stretch binary.  I have no idea why this is different with the
first run.

On my way reducing the corpus I had an intermediate corpus of
~43000 files on which notmuch new produced a xapian exeption.  I
kept this corpus but did not include the test results in my email
because the later, smaller corpus seemed more interesting.

I redid the test on this mid-sized corpus and I got an
exeption (again):

0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2820, filename=filename@entry=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len",
                                                                                              message_ret=message_ret@entry=0x7fffffffd2e8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", notmuch=0x5555557d2820) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2820, path=path@entry=0x5555557d26e0 "/tmp/reduced-sample.xapian-exeption/new", state=state@entry=0x7fffffffd560) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2820, path=path@entry=0x5555557cf730 "/tmp/reduced-sample.xapian-exeption", state=state@entry=0x7fffffffd560) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda88) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 21m 2s (21 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 6752) exited with code 01]
(gdb) 



I redid the test with this medium sized corpus with the notmuch
binary from debian stretch, the xapian exeption hits after the
same number of scanned/added files:

0 (master *) grfz@len:~$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/notmuch...(no debugging symbols found)...done.
(gdb) b _notmuch_database_log
Function "_notmuch_database_log" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_notmuch_database_log) pending.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 10m 21s (44 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 10408) exited with code 01]
(gdb) bt
No stack.
(gdb) continue
The program is not being run.
(gdb) quite
Undefined command: "quite".  Try "help".
(gdb) quit



In case your' not exhausted, the mid-sized corpus is here:
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz

Regards, Gregor

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> On my way reducing the corpus I had an intermediate corpus of
> ~43000 files on which notmuch new produced a xapian exeption.  I
> kept this corpus but did not include the test results in my email
> because the later, smaller corpus seemed more interesting.
>
> I redid the test on this mid-sized corpus and I got an
> exeption (again):
[snip]
> Error: A Xapian exception occurred. Halting processing.
> Processed 27611 total files in 21m 2s (21 files/sec.).
> Added 27481 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 6752) exited with code 01]
> (gdb) 

I can finally replicate the crash. I do crash in a different place
(after processing 25184 files), so it suggests a certain non-determinism
or dependence on the host configuration.

Anyway, now that I have a test case, I'll try and figure out what the
underlying Xapian exeception is.

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner <david@tethera.net> [14. Jul. 2017]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> I redid the test on this mid-sized corpus and I got an
>> exeption (again):
> [snip]
>> Error: A Xapian exception occurred. Halting processing.
>> Processed 27611 total files in 21m 2s (21 files/sec.).
>> Added 27481 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 6752) exited with code 01]
>> (gdb) 
> 
> I can finally replicate the crash. I do crash in a different place
> (after processing 25184 files), so it suggests a certain non-determinism
> or dependence on the host configuration.

I would assume the first difference is in the file system:
Perhaps the files are accessed in a different order!?

> Anyway, now that I have a test case, I'll try and figure out what the
> underlying Xapian exeception is.


Thanks.

Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Leonard Lausen]

I also experience this error trying to import the emacs-orgmode mailing
list archive. Has there been any progress on fixing the issue? What
steps should be taken to help?

Also it seems that the midsize corpus
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
available anymore.

David introduced a patch to improve the error message, below I paste the
message I get:

add_file: A Xapian exception occurred
A Xapian exception occurred adding message: Unexpected end of posting list for 'G00000000000255c6'.
Processed 4949 total files in 10m 13s (8 files/sec.).
Added 4921 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred


Gregor Zattler <telegraph@gmx.net> writes:

> Hi David,
> * David Bremner <david@tethera.net> [14. Jul. 2017]:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> I redid the test on this mid-sized corpus and I got an
>>> exeption (again):
>> [snip]
>>> Error: A Xapian exception occurred. Halting processing.
>>> Processed 27611 total files in 21m 2s (21 files/sec.).
>>> Added 27481 new messages to the database.
>>> Note: A fatal error was encountered: A Xapian exception occurred
>>> [Inferior 1 (process 6752) exited with code 01]
>>> (gdb) 
>> 
>> I can finally replicate the crash. I do crash in a different place
>> (after processing 25184 files), so it suggests a certain non-determinism
>> or dependence on the host configuration.
>
> I would assume the first difference is in the file system:
> Perhaps the files are accessed in a different order!?
>
>> Anyway, now that I have a test case, I'll try and figure out what the
>> underlying Xapian exeception is.
>
>
> Thanks.
>
> Ciao, Gregor
> -- 
>  -... --- .-. . -.. ..--.. ...-.-

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi Leonard, nomuch developers,
* Leonard Lausen <leonard@lausen.nl> [2018-07-01; 15:36]:
> I also experience this error trying to import the emacs-orgmode mailing
> list archive. Has there been any progress on fixing the issue? What
> steps should be taken to help?
>
> Also it seems that the midsize corpus
> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
> available anymore.

I still have this archive on my local hard drive.  If you (or
anyone else) are interested in this, I will upload it again.

Ciao; Gregor 

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi Leonard, nomuch developers,
> * Leonard Lausen <leonard@lausen.nl> [2018-07-01; 15:36]:
>> I also experience this error trying to import the emacs-orgmode mailing
>> list archive. Has there been any progress on fixing the issue? What
>> steps should be taken to help?
>>
>> Also it seems that the midsize corpus
>> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
>> available anymore.
>
> I still have this archive on my local hard drive.  If you (or
> anyone else) are interested in this, I will upload it again.
>

Please do. I seem to have deleted my copy.

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, Leonard,
* David Bremner <david@tethera.net> [2018-07-07; 23:07]:
> Gregor Zattler <telegraph@gmx.net> writes:
>> * Leonard Lausen <leonard@lausen.nl> [2018-07-01; 15:36]:
>>> Also it seems that the midsize corpus
>>> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
>>> available anymore.
>>
>> I still have this archive on my local hard drive.  If you (or
>> anyone else) are interested in this, I will upload it again.
>>
> Please do. I seem to have deleted my copy.

Done.  It's at
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz
again.

Thanks for looking into this, Gregor

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Leonard Lausen <leonard@lausen.nl> writes:

> I also experience this error trying to import the emacs-orgmode mailing
> list archive. Has there been any progress on fixing the issue? What
> steps should be taken to help?
>
> Also it seems that the midsize corpus
> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
> available anymore.
>
> David introduced a patch to improve the error message, below I paste the
> message I get:
>
> add_file: A Xapian exception occurred
> A Xapian exception occurred adding message: Unexpected end of posting list for 'G00000000000255c6'.
> Processed 4949 total files in 10m 13s (8 files/sec.).
> Added 4921 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred

While we're sorting out test data, what versions of notmuch and xapian
are you running?

d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Leonard Lausen]

I run notmuch 0.27 and xapian 1.4.3 (from the Debian stretch repo).

Now that Xapian 1.4.6 is released (which should contain fixes to prevent
the database corruption that may occur due to this bug) I will upgrade
to it.

David Bremner <david@tethera.net> writes:
> While we're sorting out test data, what versions of notmuch and xapian
> are you running?
>
> d

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Leonard Lausen <leonard@lausen.nl> writes:

> I run notmuch 0.27 and xapian 1.4.3 (from the Debian stretch repo).
>
> Now that Xapian 1.4.6 is released (which should contain fixes to prevent
> the database corruption that may occur due to this bug) I will upgrade
> to it.
>

I believe the problem is a Xapian bug still present in 1.4.6. The
following patch (to Xapian) seems to fix it for me:

          https://oligarchy.co.uk/xapian/patches/glass-cursor-rebuild-fix.patch

If I understand correctly Olly will release Xapian 1.4.7 pretty soon
with this fix (and another one that affects the notmuch test suite).

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

David Bremner <david@tethera.net> writes:

> Gregor Zattler <telegraph@gmx.net> writes:
>
>>
>> On my way reducing the corpus I had an intermediate corpus of
>> ~43000 files on which notmuch new produced a xapian exeption.  I
>> kept this corpus but did not include the test results in my email
>> because the later, smaller corpus seemed more interesting.
>>
>> I redid the test on this mid-sized corpus and I got an
>> exeption (again):
> [snip]
>> Error: A Xapian exception occurred. Halting processing.
>> Processed 27611 total files in 21m 2s (21 files/sec.).
>> Added 27481 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 6752) exited with code 01]
>> (gdb) 
>
> I can finally replicate the crash. I do crash in a different place
> (after processing 25184 files), so it suggests a certain non-determinism
> or dependence on the host configuration.
>
> Anyway, now that I have a test case, I'll try and figure out what the
> underlying Xapian exeception is.

As far as I know this was dixed in xapian 1.4.7, so marking it fixed for
notmuch.

d

confirmed (was: Bug: fatal error with notmuch new, second run starts indexing all over again)

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner <david@tethera.net> [2021-12-25; 14:10]:
> David Bremner <david@tethera.net> writes:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> On my way reducing the corpus I had an intermediate corpus of
>>> ~43000 files on which notmuch new produced a xapian exeption.  I
>>> kept this corpus but did not include the test results in my email
>>> because the later, smaller corpus seemed more interesting.
>>>
>>> I redid the test on this mid-sized corpus and I got an
>>> exeption (again):
>> [snip]
>>> Error: A Xapian exception occurred. Halting processing.
>>> Processed 27611 total files in 21m 2s (21 files/sec.).
>>> Added 27481 new messages to the database.
>>> Note: A fatal error was encountered: A Xapian exception occurred
>>> [Inferior 1 (process 6752) exited with code 01]
>>> (gdb)
>>
>> I can finally replicate the crash. I do crash in a different place
>> (after processing 25184 files), so it suggests a certain non-determinism
>> or dependence on the host configuration.
>>
>> Anyway, now that I have a test case, I'll try and figure out what the
>> underlying Xapian exeception is.
>
> As far as I know this was dixed in xapian 1.4.7, so marking it fixed for
> notmuch.

I redid the test with my mid-sized corpus from the org-mode
mailing liste which used to crash notmuch when indexing.

That's indeed not the case any more with a up to date

- debian 11/bullseye,

- GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, cairo
  version 1.16.0) of 2021-12-27,

- notmuch 0.34.2+41~g063f5e9

Thank you very much for notmuch, Gregor

Re: confirmed (was: Bug: fatal error with notmuch new, second run starts indexing all over again)

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> I redid the test with my mid-sized corpus from the org-mode
> mailing liste which used to crash notmuch when indexing.
>
> That's indeed not the case any more with a up to date
>
> - debian 11/bullseye,
>
> - GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, cairo
>   version 1.16.0) of 2021-12-27,
>
> - notmuch 0.34.2+41~
> Thank you very much for notmuch, Gregor

And thank you for confirming the bug is fixed!

d