segfault with xapian 1.3.1

segfault with xapian 1.3.1

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

As discussed on IRC, notmuch segfaults with 1.3.1 but not with 1.2.5 on
notmuch-new:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff75a5367 in Xapian::TermIterator::skip_to(std::string const&) ()
from /usr/lib64/libxapian-1.3.so.1
(gdb) bt
#0  0x00007ffff75a5367 in Xapian::TermIterator::skip_to(std::string const&)
() from /usr/lib64/libxapian-1.3.so.1
#1  0x00007ffff7decaa5 in _notmuch_message_get_term(_notmuch_message*,
Xapian::TermIterator&, Xapian::TermIterator&, char const*) () from
/usr/lib64/libnotmuch.so.3
#2  0x00007ffff7ded1fb in
_notmuch_message_ensure_metadata(_notmuch_message*) () from
/usr/lib64/libnotmuch.so.3
#3  0x00007ffff7ded535 in notmuch_message_get_message_id () from
/usr/lib64/libnotmuch.so.3
#4  0x00007ffff7de9c11 in notmuch_database_add_message () from
/usr/lib64/libnotmuch.so.3
#5  0x000000000040c610 in add_files ()
#6  0x000000000040cfbf in notmuch_new_command ()
#7  0x0000000000407a1b in main ()

This occurs irrespective of the state of the database or the message being
added.

[-- Attachment #2: Type: text/html, Size: 1242 bytes --]

Re: segfault with xapian 1.3.1

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:

> As discussed on IRC, notmuch segfaults with 1.3.1 but not with 1.2.5 on
> notmuch-new:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff75a5367 in Xapian::TermIterator::skip_to(std::string const&) ()
> from /usr/lib64/libxapian-1.3.so.1
> (gdb) bt

I managed to duplicate this problem on Debian testing by rebuilding
Olly's packages (https://launchpad.net/~ojwb/+archive/xapian-1.3) for
sid and running the following script in the ./test subdirectory

,----
| test_description='"notmuch new" segfault'
| . test-lib.sh
| 
| generate_message
| gdb notmuch <<EOF
| run new
| bt full
| EOF
`----

The backtrace is attached.


[-- Attachment #2: backtrace --]
[-- Type: text/plain, Size: 8016 bytes --]

backtrace.sh: Testing "notmuch new" segfault
GNU gdb (GDB) 7.6 (Debian 7.6-5)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/bremner/software/upstream/notmuch/notmuch...done.
(gdb) Starting program: /home/bremner/software/upstream/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff709ba37 in Xapian::TermIterator::skip_to (this=0x7fffffffdc10, term=...)
    at ../api/termiterator.cc:145
(gdb) #0  0x00007ffff709ba37 in Xapian::TermIterator::skip_to (this=0x7fffffffdc10, 
    term=...) at ../api/termiterator.cc:145
No locals.
#1  0x0000000000422959 in _notmuch_message_get_term (message=0x6833e0, i=..., 
    end=..., prefix=0x434644 "XREPLYTO") at lib/message.cc:271
        prefix_len = 8
        value = 0x6810a0 "\001"
        term = {static npos = <optimized out>, 
          _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x67eaa8 "XFDIRENTRY2:msg-001"}}
#2  0x0000000000422d1b in _notmuch_message_ensure_metadata (message=0x6833e0)
    at lib/message.cc:347
        i = {internal = 0x0}
        end = {internal = 0x0}
        thread_prefix = 0x4346a0 "G"
        replyto_prefix = 0x434644 "XREPLYTO"
        __PRETTY_FUNCTION__ = "void _notmuch_message_ensure_metadata(notmuch_message_t*)"
        tag_prefix = 0x4346a6 "K"
        id_prefix = 0x4346ae "Q"
        filename_prefix = 0x434670 "XFDIRENTRY"
#3  0x0000000000422f18 in notmuch_message_get_message_id (message=0x6833e0)
    at lib/message.cc:390
No locals.
#4  0x000000000041f14e in _notmuch_database_link_message (notmuch=0x666240, 
    message=0x6833e0, message_file=0x666140) at lib/database.cc:1676
        status = NOTMUCH_STATUS_SUCCESS
        stored_id = {static npos = <optimized out>, 
          _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
            _M_p = 0x7ffff702c3d8 <std::string::_Rep::_S_empty_rep_storage+24> ""}}
        message_id = 0x6832c0 "Tmail"
        thread_id = 0x0
        metadata_key = 0x67ea80 ""
#5  0x000000000041f769 in notmuch_database_add_message (notmuch=0x666240, 
    filename=0x666090 "/home/bremner/software/upstream/notmuch/test/tmp.backtrace/mail/msg-001", message_ret=0x7fffffffdee8) at lib/database.cc:1837
        message_file = 0x666140
        message = 0x6833e0
        private_status = NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND
        from = 0x67ee00 "Notmuch Test Suite <test_suite@notmuchmail.org>"
        ret = NOTMUCH_STATUS_SUCCESS
        ret2 = NOTMUCH_STATUS_SUCCESS
        to = 0x67eec0 "Notmuch Test Suite <test_suite@notmuchmail.org>"
        message_id = 0x67eaa0 "\001"
        date = 0x0
        header = 0x67ef10 "<msg-001@notmuch-test-suite>"
        subject = 0x6823c0 "Test message #1"
#6  0x000000000040e91c in add_files (notmuch=0x666240, 
    path=0x664a80 "/home/bremner/software/upstream/notmuch/test/tmp.backtrace/mail", state=0x7fffffffe1d0) at notmuch-new.c:507
        dir = 0x0
        entry = 0x665d50
        next = 0x666090 "/home/bremner/software/upstream/notmuch/test/tmp.backtrace/mail/msg-001"
        fs_mtime = 1378426382
        db_mtime = 0
        status = NOTMUCH_STATUS_SUCCESS
        ret = NOTMUCH_STATUS_SUCCESS
        message = 0x0
        fs_entries = 0x6811d0
        i = 2
        num_fs_entries = 4
        entry_type = 32768
        directory = 0x0
        db_files = 0x0
        db_subdirs = 0x0
        stat_time = 1378426382
        st = {st_dev = 65031, st_ino = 3451162, st_nlink = 3, st_mode = 16877, 
          st_uid = 1000, st_gid = 1000, __pad0 = 0, st_rdev = 0, st_size = 4096, 
          st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1378426382, 
            tv_nsec = 0}, st_mtim = {tv_sec = 1378426382, tv_nsec = 0}, st_ctim = {
            tv_sec = 1378426382, tv_nsec = 0}, __unused = {0, 0, 0}}
        is_maildir = 0
        tag = 0x7fffffffe480
#7  0x000000000040fa12 in notmuch_new_command (config=0x661eb0, argc=1, 
    argv=0x7fffffffe490) at notmuch-new.c:949
        notmuch = 0x666240
        add_files_state = {output_is_a_tty = 0, verbose = 0, debug = 0, 
          new_tags = 0x665ef0, new_tags_length = 2, new_ignore = 0x665fe0, 
          new_ignore_length = 0, total_files = 1, processed_files = 1, 
          added_messages = 0, removed_messages = 0, renamed_messages = 0, 
          tv_start = {tv_sec = 1378426382, tv_usec = 768704}, 
          removed_files = 0x681a10, removed_directories = 0x665d30, 
          directory_mtimes = 0x665c00, synchronize_flags = 1}
        elapsed = 2.1219957909652723e-314
        tv_now = {tv_sec = 0, tv_usec = 0}
        tv_start = {tv_sec = 0, tv_usec = 140737488348288}
        ret = 0
        st = {st_dev = 4294967295, st_ino = 140737353968080, 
          st_nlink = 140737328394192, st_mode = 4160580048, st_uid = 32767, 
          st_gid = 160, __pad0 = 0, st_rdev = 67, st_size = 128, st_blksize = 0, 
          st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {
            tv_sec = 210453397510, tv_nsec = 472446402651}, st_ctim = {
            tv_sec = 532575944823, tv_nsec = 0}, __unused = {6546568, 0, 6548008}}
        db_path = 0x664a80 "/home/bremner/software/upstream/notmuch/test/tmp.backtrace/mail"
        dot_notmuch_path = 0x0
        action = {__sigaction_handler = {sa_handler = 0x40db03 <handle_sigint>, 
            sa_sigaction = 0x40db03 <handle_sigint>}, sa_mask = {__val = {
              0 <repeats 16 times>}}, sa_flags = 268435456, sa_restorer = 0x0}
        f = 0x7fffffffe480
        opt_index = 1
        i = 32767
        timer_is_active = 0
        no_hooks = 0
        options = {{opt_type = NOTMUCH_OPT_BOOLEAN, output_var = 0x7fffffffe1d4, 
            name = 0x432339 "verbose", arg_id = 118, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_BOOLEAN, output_var = 0x7fffffffe1d8, 
            name = 0x432341 "debug", arg_id = 100, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_BOOLEAN, output_var = 0x7fffffffe07c, 
            name = 0x432347 "no-hooks", arg_id = 110, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_END, output_var = 0x0, name = 0x0, arg_id = 0, 
            keywords = 0x0}}
#8  0x000000000040a519 in main (argc=2, argv=0x7fffffffe488) at notmuch.c:303
        local = 0x64b090
        talloc_report = 0x42fcfd <__libc_csu_init+93> "H\203\303\001H9\353u\352H\213\\$\bH\213l$\020L\213d$\030L\213l$ L\213t$(L\213|$0H\203\304\070\303\017\037\200"
        command_name = 0x7fffffffe7d1 "new"
        command = 0x63ee00 <commands+64>
        config_file_name = 0x0
        config = 0x661eb0
        print_help = 0
        print_version = 0
        opt_index = 1
        ret = 0
        options = {{opt_type = NOTMUCH_OPT_BOOLEAN, output_var = 0x7fffffffe354, 
            name = 0x4303a7 "help", arg_id = 104, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_BOOLEAN, output_var = 0x7fffffffe350, 
            name = 0x430c99 "version", arg_id = 118, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_STRING, output_var = 0x7fffffffe358, 
            name = 0x430368 "config", arg_id = 99, keywords = 0x0}, {
            opt_type = NOTMUCH_OPT_END, output_var = 0x0, name = 0x0, arg_id = 0, 
            keywords = 0x0}}
(gdb) quit
A debugging session is active.

	Inferior 1 [process 30499] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]
 FATAL  

Unexpected exit while executing ./backtrace.sh. Exit code 0.

Re: segfault with xapian 1.3.1

[David Bremner]

David Bremner <david@tethera.net> writes:

>
> I managed to duplicate this problem on Debian testing by rebuilding
> Olly's packages (https://launchpad.net/~ojwb/+archive/xapian-1.3) for
> sid and running the following script in the ./test subdirectory

Olly wrote on  IRC:

> bremner: ok, 1.2 explicitly no-oped skip_to() on an iterator at_end on
> trunk that code has been rewritten without that explicit check, and
> the iterator internals are NULL then i think restoring the check is
> reasonable, though I'm not sure if we actually promise that's defined
> behaviour if you want to work with 1.3.1, then checking against the
> end iterator before calling skip_to() will work for any version

So, now we know what to fix.

d

Re: segfault with xapian 1.3.1

[Olly Betts]

Olly wrote on  IRC:
> bremner: ok, 1.2 explicitly no-oped skip_to() on an iterator at_end on
> trunk that code has been rewritten without that explicit check, and
> the iterator internals are NULL then i think restoring the check is
> reasonable, though I'm not sure if we actually promise that's defined
> behaviour if you want to work with 1.3.1, then checking against the
> end iterator before calling skip_to() will work for any version

I've put a NULL check in there, and also for the other iterator classes
which use NULL internals to signify being at the end - this only worked
for TermIterator in 1.2.x, but it seems reasonable to make it work in
general and a NULL pointer check isn't a big overhead.

On Thu, Sep 05, 2013 at 10:22:42PM -0300, David Bremner wrote:
> So, now we know what to fix.

This should work again in 1.3.2 (once it is out), but if you want to
support 1.3.1 then you need the check.

Cheers,
    Olly

Re: segfault with xapian 1.3.1

[David Bremner]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:

> As discussed on IRC, notmuch segfaults with 1.3.1 but not with 1.2.5 on
> notmuch-new:
>

Ahem. Years later, and notmuch works OK with xapian 1.3.4, so I think we
can consider this bug closed.

d