notmuch release 0.4 now available

notmuch release 0.4 now available

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 11069 bytes --]

After a (regrettable) 6-month hiatus, here is a new release of notmuch.

I have not yet completed the merging of all interesting, outstanding
patches, (nor even all the obvious ones), but I felt it was important to
start having releases happen again.

It may sound foolhardy, but I really would like to have releases happen
as often as once per week. My plan is to start each week by simply
looking to see if new code has landed, and if so, push it out in a new
release.

Let's see how well that works.

In the meantime, enjoy this release which provides some important new
command-line functionality, (notmuch search --output, notmuch show
--format=mbox), lots of emacs interfaces (all sent messages now saved to
mail store by default), and one fairly critical bug fix (avoid a
possibility for a corrupt database if "notmuch new" is interrupted).

There's a lot more beyond that as well. See below for details.

-Carl

-- 
carl.d.worth@intel.com

Where to obtain notmuch 0.4
===========================
  http://notmuchmail.org/releases/notmuch-0.4.tar.gz

Which can be verified with:

  http://notmuchmail.org/releases/notmuch-0.4.tar.gz.sha1
  d40cb07adab0d595c64e9b73f1fd306cc9dff106  notmuch-0.4.tar.gz

  http://notmuchmail.org/releases/notmuch-0.4.tar.gz.sha1.asc
  (signed by Carl Worth)

What's new in notmuch 0.4
=========================
New command-line features
-------------------------
notmuch search --output=(summary|threads|messages|tags|files)

  This new option allows for particular items to be returned from
  notmuch searches. The "summary" option is the default and behaves
  just as "notmuch search" has historically behaved.

  The new option values allow for thread IDs, message IDs, lists of
  tags, and lists of filenames to be returned from searches. It is
  expected that this new option will be very useful in shell
  scripts. For example:

	for file in $(notmuch search --output=files <search-terms>); do
		<operations-on> "$file"
	done

notmuch show --format=mbox <search-specification>

  This new option allows for the messages matching a search
  specification to be presented as an mbox. Specifically the "mboxrd"
  format is used which allows for reversible quoting of lines
  beginning with "From ". A reader should remove a single '>' from the
  beginning of all lines beginning with one or more '>' characters
  followed by the 5 characters "From ".

notmuch config [get|set] <section>.<item> [value ...]

  The new top-level "config" command allows for any value in the
  notmuch configuration file to be queried or set to a new value. Both
  single-valued and multi-valued items are supported, as our any
  custom items stored in the configuration file.

Avoid setting Bcc header in "notmuch reply"

  We decided that this was a bit heavy-handed as the actual mail
  user-agent should be responsible for setting any Bcc option. Also,
  see below for the notmuch/emacs user-agent now setting an Fcc by
  default rather than Bcc.

New library features
--------------------
Add notmuch_query_get_query_string and notmuch_query_get_sort

  These are simply functions for querying properties of a
  notmuch_query_t object.

New emacs features
------------------
Enable Fcc of all sent messages by default (to "sent" directory)

  All messages sent from the emacs interface will now be saved to the
  notmuch mail store where they will be incorporated to the database
  by the next "notmuch new". By default, messages are saved to the
  "sent" directory at the top-level of the mail store. This directory
  can be customized by means of the "Notmuch Fcc Dirs" option in the
  notmuch customize interface.

Ability to all open messages in a thread to a pipe

  Historically, the '|' keybinding allows for piping a single message
  to an external command. Now, by prefixing this key with a prefix
  argument, (for example, by pressing "Control-U |"), all open
  messages in the current thread will be sent to the external command.

Optional support for detecting inline patches

  This hook is disabled by default but can be enabled with a checkbox
  under "Notmuch Show Insert Text/Plain Hook" in the notmuch customize
  interface. It allows for inline patches to be detected and treated
  as if they were attachments, (with context-sensitive highlighting).

Automatically tag messages as "replied" when sending a reply

  Messages replied to within the emacs interface will now be tagged as
  "replied". This feature can easily be customized to add or remove
  other tags as well. For example, a user might use a tag of
  "needs-reply" and can configure this feature to automatically remove
  that tag when replying. See "Notmuch Message Mark Replied" in the
  notmuch customize interface.

Allow search-result color specifications to overlay each other

  For example, one tag can specify the background color of matching
  lines, while another can specify the foreground. With this change,
  both settings will now be visible simultaneously, (which was not the
  case in previous releases). See "Notmuch Search Line Faces" in the
  notmuch customize interface.

Make hidden author names still available for incremental search.

  When there is insufficient space to display all authors of a thread
  in search results, the names of hidden authors are now still made
  available to emacs' incremental search commands. As the user
  searches, matching lines will temporarily expand to show the hidden
  names.

New binding of Control-TAB (works like TAB in reverse)

  Many notmuch nodes already use TAB to navigate forward through
  various items allowing actions, (message headers, email attachments,
  etc.). The new Control-TAB binding operates similarly but in the
  opposite direction.

New build-system features
-------------------------
Various portability fixes have been applied

  These include fixes for build failures on at least Solaris, FreeBSD,
  and Fedora systems. We're hopeful that the notmuch code base is now
  more portable than ever before.

Arrange for libnotmuch to be found automatically after make install

  The notmuch build system is now careful to help the user avoid
  errors of the form "libnotmuch.so could not be found" immediately
  after installing. This support takes two forms:

	1. If the library is installed to a system directory,
	   (configured in /etc/ld.so.conf), then "make install" will
	   automatically run ldconfig.

	2. If the library is installed to a non-system directory, the
	   build system adds a DR_RUNPATH entry to the final binary
	   pointing to the directory to which the library is installed.

  When this support works, the user should be able to run notmuch
  immediately after "make install", without any errors trying to find
  the notmuch library, and without having to manually set environment
  variables such as LD_LIBRARY_PATH.

Check compiler/linker options before using them

  The configure script now carefully checks that any desired
  compilation options, (whether for enabling compiler warnings, or for
  embedding rpath, etc.), are supported. Only supported options are
  used in the resulting Makefile.

New test-suite features
-----------------------
New modularization of test suite.

  Thanks to a gracious relicensing of the test-suite infrastructure
  from the git project, notmuch now has a modular test suite. This
  provides the ability to run individual sections of the test suite
  rather than the whole things. It also provides better summary of
  test results, with support for tests that are expected to fail
  (BROKEN and FIXED) in addition to PASS and FAIL. Finally, it makes
  it easy to run the test suite within valgrind (pass --valgrind to
  notmuch-test or to any sub-script) which has been very useful.

New testing of emacs interface.

  The test suite has been augmented to allow automated testing of the
  emacs interfaces. So far, this includes basic searches, display of
  threads, and tag manipulation. This also includes a test that a new
  message can successfully be sent out through a (dummy) SMTP server
  and that said message is successfully integrated into the notmuch
  database via the FCC setting.

General bug fixes
-----------------
Fix potential corruption of database when "notmuch new " is interrupted.

  Previously, an interruption of "notmuch new" would (rarely) result
  in a corrupt database. The corruption would manifest itself by a
  persistent error of the form:

	document ID of 1234 has no thread ID

  The message-adding code has been carefully audited and reworked to
  avoid this sort of corruption regardless of when it is interrupted.

Fix failure with extremely long message ID headers.

  Previously, a message with an extremely long message ID, (say, more
  than 300 characters), would fail to be added to notmuch, (triggering
  Xapian exceptions). This has now been fixed.

Fix for messages with "charset=unknown-8bit"

  Previously, messages with this charset would cause notmuch to emit a
  GMime warning, (which would then trip up emacs or other interfaces
  parsing the notmuch results).

Fix notmuch_query_search_threads function to return NULL on any exception

Fix "notmuch search" to return non-zero if notmuch_query_search_threads fails

  Previously, this command could confusingly report a Xapian
  exception, yet still return an error code of 0. It now correctly
  returns a failing error code of 1 in this case.

Emacs bug fixes
---------------
Fix to handle a message with a subject containing, for example "[1234]"

  Previously, a message subject containing a sequence of digits within
  square brackets would cause the emacs interface to mis-parse the
  output of "notmuch search". This would result in the message being
  mis-displayed and prevent the user from manipulating the message in
  the emacs interface.

Fix to correctly handle message IDs containing ".."

  The emacs interface now properly quotes message IDs to avoid a
  Xapian bug in which the ".." within a message ID would be
  misinterpreted as a numeric range specification.

Python-binding fixes
--------------------
The python bindings for notmuch have been updated to work with python3.

Debian-specific fixes
---------------------
Fix emacs initialization so "M-x notmuch" works for users by default.

  Now, a new Debian user can immediately run "emacs -f notmuch" after
  "apt-get install notmuch". Previously, the user would have had to
  edit the ~/.emacs file to add "(require 'notmuch)" before this would
  work.

What is notmuch
===============
Notmuch is a system for indexing, searching, reading, and tagging
large collections of email messages in maildir or mh format. It uses
the Xapian library to provide fast, full-text search with a convenient
search syntax.

For more about notmuch, see http://notmuchmail.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

***UNCHECKED*** bug report: notmuch-hello 'All tags' view

[Darren McGuicken]

[-- Attachment #1: Type: application/pgp-encrypted, Size: 11 bytes --]

[-- Attachment #2: Type: application/octet-stream, Size: 1729 bytes --]

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

hQEOA76aKcGOPNjWEAP+NfuUZr2YjRjLwEgAqQgqlcPc2RFo2DA9gPzq6moCOg9I
J2nRf2xrje5u/LPz7bnrDoA16bClnlvN3P7zLfqIwt7i12F4YUlyjE+a3IqtwNQU
yGsQis97piMmhZDhjEhAMhmEeHZlnedZT7CzbRA7Fcjxl1ehnoflMOp3YWPwIy4E
AKFVQ1/EcCiCafkzRpLO34jixmSCCG/wHYG6Qje3yBURJ5Ir5TV+D3GP3Z76u+c9
3nUS3k/bQE4zmOkcE5DpyJGJtJLPMXsFJTwvS4zLtnjbUE14hAbkai2fMB17GCs4
0lT+sVM8jHOIr+ef3gkc73z1xePV5q0WeXGs/HHOfl8F0ukBu+Ojt0tGzUqKo9Pw
pKORwTr43v6pWCU2Fh744vfjfvc+HpT2YjGL0PxU7/vZLD2VUBa59vKUAHevr/Mq
NmXS2vlo6ORfkgrTfEXgntWBFNoepcs9l/PXXrXQQDVgCTR6S70RhoSy1vjb4VTz
lM/0zKj51HhSnjCFKAgqM6GuF6TYxsrBO9lGg3pwfR5fAmpYKADFqAVl3gIU4Cv/
30tgmA9aKsd+zRZKi2VlNtdXI2WapsGgfKuPBm/P7Xti2EFjxen05OHz77z9pDLq
/ImQVp4Gh3AoKIZehxyi5j3M6ff91EGbjkDqkPYGNIMY/389QdDECNZuDg8h++4Y
zD3nBDAKeVC7bh6gXL6GQPYM/Me/IezTtLbBJXhoKBDsmU4P0zmykv9IRsG+sTZ4
+JvGDDT19TYnx0lXRJowI+7BLbVFXEkY7H++VPpmUSFnhWilOwLD3JNt4dmLV54n
NLjiRBKCNKNTUzUwMJ93/WjJy3+LYObWdzLatx989uhFQVVSSxNHjeDA/69/xMHi
qNRANl/cQCwNcPriZqH+ZewKPm/gQKzth5NFHnAFEZenzp/8Vt7kAJFN+m0gH3ew
C+ij8WMHwOLckuBMi1P6DWKltzDNhkxXzlbkmfvYVxCZOZZ3fvXXVF1Z4/Xo+ljZ
clZ+Rj0EJiZeuByJw0oUSLxpdcDgvoRdrjpqwD75eadlpSmVq+0eXfyf90Pfkqjf
dceA9uCzZEYYKJa6T6eh1l1N/ULYfy/8Ppk1ZL0kbLjklZngW6G1QOYdaK7kCaFl
GCwJtoaCsctQx87iUD4bUOnEmD+WJ16YRVfaqwPYrSFwFKdE7fK3K9IrpX+s07zp
1oz4eCJw8EbyqqmCcuuDKJRpUbb8feT12sNA9Q6M5AInzUF7yyyETXg2A/zCwGmo
S0YEamw9mDII8U/aTrmAVpa4WjOi/UvDxe7KC2/pEY3+qwWZTaX2qN5VutTHZRwT
/S/S/WHsJOKoPPBszY3Bk1s8AnSm9r4uK5/I0cvrqpO6JagG0faQnmRuuSVyoE78
GFv7MPv9A0TM6YnCAot8DEWCNCxpwMQlcW/HY+8teQybokyj165cZffBFlAl76dk
xk5LCz3BbfVnMv/qAEdDplhzqJQbMOpzyuDIQ8ma7UXwF/sI9mc9pItwXw8AOTlu
UP9Q79M6WwUgEXTgy8XcL0tCGCeNBeQ/3ML4cbD7LqOIyC+xDg8moiKSoWfAOxgr
icQG/50=
=BakF
-----END PGP MESSAGE-----

Re: bug report: notmuch-hello 'All tags' view

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Wow, that was probably the most secure bug report in history - this time
in plain for those of you who don't have access to Carl's private key!

I've noticed since rebasing to 0.4 that I'm seeing an empty entry in the
'All tags' view of notmuch-hello which appears to represent a search
against 'tag:'.  

For some reason this search matches 1,233 of my now 22,905 messages,
encompassing a date range of about five years and including an
apparently random collection of both tagged and untagged threads.

Anyone else seeing anything similar?  What else do you need from me to
track this one down?

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch release 0.4 now available

[Scott Henson]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

On Tue, Nov 2, 2010 at 2:33 AM, Carl Worth <cworth@cworth.org> wrote:

> Where to obtain notmuch 0.4
> ===========================
>  http://notmuchmail.org/releases/notmuch-0.4.tar.gz
>
> Which can be verified with:
>
>  http://notmuchmail.org/releases/notmuch-0.4.tar.gz.sha1
>  d40cb07adab0d595c64e9b73f1fd306cc9dff106  notmuch-0.4.tar.gz
>
>  http://notmuchmail.org/releases/notmuch-0.4.tar.gz.sha1.asc
>  (signed by Carl Worth)
>
>
I've built 0.4 for Fedora in case anyone is interested.  Currently, it is
only built for fc13 x86_64, but I'll likely be building for fc14 x86_64 as
soon as I get fc14 installed.  If anyone wants i686, I can probably be
convinced to spin up a vm to do that in.

http://aiden.foolishpride.org/~shenson/repos/fc13/
http://aiden.foolishpride.org/~shenson/repos/fc13-sources/

I'll also likely be submitting it to Fedora proper as I have time.  Thanks
everyone for all the work!


-- 
Scott Henson

[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

Re: bug report: notmuch-hello 'All tags' view

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On Tue, 02 Nov 2010 20:17:56 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> I've noticed since rebasing to 0.4 that I'm seeing an empty entry in the
> 'All tags' view of notmuch-hello which appears to represent a search
> against 'tag:'.  

Continuing what's turning into an epic one-man list-noise generation
effort, I did a bit more digging and it looks like notmuch-hello
generates the list of tags for the 'All tags' view by processing each
line returned by 'search-tags'.

A call to 'notmuch search-tags' from the command line does indeed return
an empty string as the first entry for me[1].

Does that point to bad data on my part, or has the output of
'search-tags' changed recently?

If the former, any thoughts on how I differentiate an empty tag from
something untagged in a search so I can prune it?


[1] So we have an apparently empty tag being concatenated with a search
    string of 'tag:', which actually gives us a search for every message
    containing the word 'tag' rather than a tag-based search.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: bug report: notmuch-hello 'All tags' view

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

On Tue, 02 Nov 2010 21:03:55 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> A call to 'notmuch search-tags' from the command line does indeed return
> an empty string as the first entry for me[1].
> 
> Does that point to bad data on my part, or has the output of
> 'search-tags' changed recently?

fwiw, I don't personally get any empty strings in the output of
search-tags (with 0.4).  So I don't think it's anything inherent to that
function.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch release 0.4 now available

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Tue, 2 Nov 2010 12:09:29 -0400, Scott Henson <sjh@foolishpride.org> wrote:
> I've built 0.4 for Fedora in case anyone is interested.  Currently, it is
> only built for fc13 x86_64, but I'll likely be building for fc14 x86_64 as
> soon as I get fc14 installed.  If anyone wants i686, I can probably be
> convinced to spin up a vm to do that in.
> 
> http://aiden.foolishpride.org/~shenson/repos/fc13/
> http://aiden.foolishpride.org/~shenson/repos/fc13-sources/

Thanks for sharing those, Scott.

> I'll also likely be submitting it to Fedora proper as I have time.  Thanks
> everyone for all the work!

I didn't mention it, but I've got the notmuch release process pushing
things directly into Debian. If there's anything I can do to help with
Fedora packaging I'd be happy to do that too.

We do have a notmuch.spec file in the tree already. Is it up to date?
And I was all setup as a Fedora contributor at one point. I might be
able to resurrect that. Would it be useful for the same "make release"
that I run to also do something with Fedora down the road? If someone
would like to set that up, I'd be happy to have that happen.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: bug report: notmuch-hello 'All tags' view

[Darren McGuicken]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

On Tue, 02 Nov 2010 17:33:08 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Tue, 02 Nov 2010 21:03:55 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> > A call to 'notmuch search-tags' from the command line does indeed
> > return an empty string as the first entry for me[1].
> 
> fwiw, I don't personally get any empty strings in the output of
> search-tags (with 0.4).  So I don't think it's anything inherent to
> that function.

Thanks for the confirmation Jamie - I had a potter with my data and sure
enough, a search for 'tag:""' returned one hit.  I was able to use the
emacs interface to remove the empty string tag from the matching message
('-' then return with no parameters).

So it's probably fair to say that we'll want to handle at least the
empty string scenario as part of our standard tag addition logic since
it can lead to unexpected side effects in searching.

Do we have any other 'known bad' scenarios in tag addition that we want
to catch before I try my hand at a patch?

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: bug report: notmuch-hello 'All tags' view

[Michal Sojka]

On Tue, 02 Nov 2010 20:17:56 +0000, Darren McGuicken <mailing-notmuch@fernseed.info> wrote:
> Wow, that was probably the most secure bug report in history - this time
> in plain for those of you who don't have access to Carl's private key!
> 
> I've noticed since rebasing to 0.4 that I'm seeing an empty entry in the
> 'All tags' view of notmuch-hello which appears to represent a search
> against 'tag:'.  
> 
> For some reason this search matches 1,233 of my now 22,905 messages,
> encompassing a date range of about five years and including an
> apparently random collection of both tagged and untagged threads.
> 
> Anyone else seeing anything similar?  What else do you need from me to
> track this one down?

I can confirm this with 0.4 debian package. notmuch search-tags outputs
first line as empty and emacs shows it in All tags view.

-Michal

Re: bug report: notmuch-hello 'All tags' view

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

On Wed, 03 Nov 2010 22:57:55 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> I can confirm this with 0.4 debian package. notmuch search-tags outputs
> first line as empty and emacs shows it in All tags view.

Again, this is *not* what I am seeing.  Michael, are you sure you don't
also have messages tagged with the empty string or blank spaces, as did
Darren?

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch release 0.4 now available

[Scott Henson]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

On Tue, Nov 2, 2010 at 5:49 PM, Carl Worth <cworth@cworth.org> wrote:
>
> > I'll also likely be submitting it to Fedora proper as I have time.
>  Thanks
> > everyone for all the work!
>
> I didn't mention it, but I've got the notmuch release process pushing
> things directly into Debian. If there's anything I can do to help with
> Fedora packaging I'd be happy to do that too.
>
>
I can have the package created and then add you to the packaging group.  I
figure that will be easier to not require you going through the review
process.



> We do have a notmuch.spec file in the tree already. Is it up to date?
> And I was all setup as a Fedora contributor at one point. I might be
> able to resurrect that. Would it be useful for the same "make release"
> that I run to also do something with Fedora down the road? If someone
> would like to set that up, I'd be happy to have that happen.
>

The spec file in the tree was a good starting place, but I had to modify it
quite a bit to get it to build properly.  I've got one version for building
from git and one version for building a release.  The release one is in the
src rpm I previously listed and I'll send a patch to the list tomorrow for
the git version.

-- 
Scott Henson

[-- Attachment #2: Type: text/html, Size: 1811 bytes --]

Re: bug report: notmuch-hello 'All tags' view

[Michal Sojka]

On Wed, 03 Nov 2010, Jameson Rollins wrote:
> On Wed, 03 Nov 2010 22:57:55 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > I can confirm this with 0.4 debian package. notmuch search-tags outputs
> > first line as empty and emacs shows it in All tags view.
> 
> Again, this is *not* what I am seeing.  Michael, are you sure you don't
> also have messages tagged with the empty string or blank spaces, as did
> Darren?

Thanks, you are probably right. I should have read your previous mail more
carefully. This was probably caused my initial tagging script which was
fooled by the bug I reported recently in
id:"1288628140-15410-1-git-send-email-sojkam1@fel.cvut.cz".

-Michal

Re: notmuch release 0.4 now available

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Wed, 3 Nov 2010 21:21:41 -0400, Scott Henson <scott@foolishpride.org> wrote:
> I can have the package created and then add you to the packaging group.  I
> figure that will be easier to not require you going through the review
> process.

Sounds good.

> The spec file in the tree was a good starting place, but I had to modify it
> quite a bit to get it to build properly.  I've got one version for building
> from git and one version for building a release.  The release one is in the
> src rpm I previously listed and I'll send a patch to the list tomorrow for
> the git version.

Excellent. I'll look forward to all of your future contributions.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]