[BUG] Emacs UI dropping every 25th line, roughly

[BUG] Emacs UI dropping every 25th line, roughly

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 2293 bytes --]

Hallo!

I noticed this one on a system with my work emails (which is what I
recently reported on the IRC channel), but can reproduce it on another
system, too.  :-/

The first system is using Ubuntu's emacs23 23.1+1-4ubuntu7.1+maverick1,
the other Debian's emacs23 23.2+1-7.

In the Emacs UI, do a ``M-x notmuch-seach RET tag:notmuch RET'', wait for
it to finish and save the buffer's text to a file, called emacs.

(In my DB, the notmuch mailing list is tagged as `notmuch' -- but it
doesn't matter, just use a search term that matches some hundreds of
messages; ``*'' if you want.)

    $ notmuch search tag:notmuch > stdout
    $ diff -w -U1 \
        <(sed < stdout -e 's%^thread:[0-9a-f]*\ %%' -e 's%|%,%' -e 's%;%%' ) \
        emacs \
        | cut -c -30
    --- /dev/fd/63	2011-01-30 21:4
    +++ emacs	2011-01-30 21:30:39.
    @@ -76,3 +76,2 @@
      November 21 [1/1] Dmitry Kur
    - November 21 [9/9] Tassilo Ho
      November 20 [5/5] Matthieu L
    @@ -100,3 +99,2 @@
      November 12 [9/9] Jameson Ro
    - November 12 [3/3] Jameson Ro
      November 12 [7/7] David Edmo
    @@ -125,3 +123,2 @@
      November 03 [13/13] david@te
    - November 02 [2/2] Gregor Kap
      November 01 [10/10] Felipe C
    @@ -151,3 +148,2 @@
       October 13 [11/11] Michal S
    -  October 13 [14/14] Felipe C
       October 11 [1/1] Kristoffer
    @@ -177,3 +173,2 @@
       2010-07-06 [2/2] dbp@riseup
    -  2010-07-05 [9/9] Dmitry Kur
       2010-06-27 [1/1] servilio@g
    @@ -203,3 +198,2 @@
       2010-06-04 [2/2] Tomas Carn
    -  2010-06-04 [2/2] dme@dme.or
       2010-06-04 [4/4] Sebastian 
    [...]

That is, roughly every 25th line is dropped in the Emacs notmuch-search
buffer!  (Via setting notmuch-command to a shell script, I intercepted
the notmuch / emacs pipeline with tee, and found that what comes out of
``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
notmuch elisp code issue.)

And, 25 times the medium length of a ``notmuch search [...]'' output line
is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
notmuch elisp code, or is Emacs doing something awful with (wild
speculation...) short reads on buffer page boundaries (or whatever else)?


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [BUG] Emacs UI dropping every 25th line, roughly

[Jesse Rosenthal]

Hi Thomas,

On Sun, 30 Jan 2011 22:02:03 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> Hallo!
> 
> I noticed this one on a system with my work emails (which is what I
> recently reported on the IRC channel), but can reproduce it on another
> system, too.  :-/

I notice this as well, reliably, when I pipe it through a shell script
(to use it over ssh) -- except in my case it's every 10 or so. When I
tell the shell to sleep 0.02 between each message it's fine. 0.01
doesn't quite do it -- whatever "it" is. Not that that's a solution
(more like throwing darts blindfolded) but it might help point someone
in the right direction.

By the way, as I mentioned on IRC a while back, when I was playing
around with this, I discovered that I could get the emacs UI to skip
every nth message just by having "notmuch search" cat a text file of
search results. That could make testing easier.

Best,
Jesse

Re: [BUG] Emacs UI dropping every 25th line, roughly

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

Hallo!

On Sun, 30 Jan 2011 22:02:03 +0100, I wrote:
> That is, roughly every 25th line is dropped in the Emacs notmuch-search
> buffer!  (Via setting notmuch-command to a shell script, I intercepted
> the notmuch / emacs pipeline with tee, and found that what comes out of
> ``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
> notmuch elisp code issue.)
> 
> And, 25 times the medium length of a ``notmuch search [...]'' output line
> is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
> notmuch elisp code, or is Emacs doing something awful with (wild
> speculation...) short reads on buffer page boundaries (or whatever else)?

I began to analyze this.  Here is a dump of my steps.

I used strace on the (already running) Emacs process.  strace would log
to several files: one for the main (Emacs) process, and a new one for
each forked process.  I can see that a ``notmuch search [...]'' process
is fork()ed / exec()uted.  It does a lot of read()ing from the DB, and
write()s out the results record by record (which in our case means line
by line).  Up to the point where I examined, there have been no short
writes.  The log:

    [some write()s for the first two dozen search results]
    16:26:54.143464 write(1, "thread:00000000000006c7   2009-12-16 [2/2] Thomas Schwinge; [subject] ([flags])\n", 135) = 135
    [...]
    16:26:54.266866 write(1, "thread:0000000000000ac4   2009-12-16 [1/1] jsm28@sourceware.org; [subject] ([flags])\n", 181) = 181

At this / in the middle of this point, the 4 KiB size is hit.  After
accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
first chunk (line breaks inserted for clarity):

    16:26:57.928798 read(8, "[first two dozen results]"
                            "thread:00000000000006c7   2009-12-16 [2/2] Thomas Schwinge; [subject] ([flags])\n"
                            "t", 4096) = 4095

The last result line is obviously incomplete, only the `t' so far.

There is more to be read, so Emacs quickly goes on with the next chunk:

    16:26:57.966247 read(8, "hread:0000000000000ac4   2009-12-16 [1/1] jsm28@sourceware.org; [subject] ([flags])\n"
                            "[more results]", 4096) = 4095

That's the remainder of the partial line, plus further results.  This is
all fine.  It's now Emacs' job to re-assemble the two partial lines
(thread ac4).  I note that indeed this search result is missing in the
notmuch-search buffer.

(By the way, I wonder why read() returns 4095 bytes instead of 4096?)

I'll next move on to debugging what actually appears at the elisp layer.
I somehow don't think the Emacs core is faulty; probably rather our
asynchronous results presentation layer handles partial lines
incorrectly?  More to come later.


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [BUG] Emacs UI dropping every 25th line, roughly

[Thomas Schwinge]

Hallo!

Here is the problem, as suspected.

On Wed, 02 Feb 2011 17:12:16 +0100, I wrote:
> At this / in the middle of this point, the 4 KiB size is hit.  After
> accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
> first chunk (line breaks inserted for clarity):
> 
>     16:26:57.928798 read(8, "[first two dozen results]"
>                             "thread:00000000000006c7   2009-12-16 [2/2] Thomas Schwinge; [subject] ([flags])\n"
>                             "t", 4096) = 4095
> 
> The last result line is obviously incomplete, only the `t' so far.

notmuch.el:notmuch-search-process-filter will be called for processing
these lines.  It'll do fine up to the single `t' -- which simply isn't a
conforming line and thus will be dropped (which is questionable behavior
anyway, I would think?).

> There is more to be read, so Emacs quickly goes on with the next chunk:
> 
>     16:26:57.966247 read(8, "hread:0000000000000ac4   2009-12-16 [1/1] jsm28@sourceware.org; [subject] ([flags])\n"
>                             "[more results]", 4096) = 4095

Same thing; this time ``hread:[...]'' isn't a confirming line, and will
be dropped.  After that, regular processing continues.


This problem has been around as of the beginning of the incremental /
asynchronous search results changes, as documented in
id:87aayatnw4.fsf@yoom.home.cworth.org on 2009-11-25; 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.


Emacs LISP is not my speciality, neither is any other LISP dialect, so
someone please carefully review this.


Grüße (und gute Nacht...),
 Thomas

[PATCH 1/2] New test: Emacs' forgetfulness.

[Thomas Schwinge]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

Signed-off-by: Thomas Schwinge <thomas@schwinge.name>
---
 test/emacs-forgetfulness |   38 ++++++++++++++++++++++++++++++++++++++
 test/notmuch-test        |    1 +
 2 files changed, 39 insertions(+), 0 deletions(-)
 create mode 100755 test/emacs-forgetfulness

diff --git a/test/emacs-forgetfulness b/test/emacs-forgetfulness
new file mode 100755
index 0000000..e17b26f
--- /dev/null
+++ b/test/emacs-forgetfulness
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+test_description=Emacs\'\ forgetfulness
+
+. test-lib.sh
+
+# RFC822 imposes a 998 character limit per line.
+x=0123456789 # 10
+x=$x$x$x$x$x$x$x$x$x$x # 100
+x=$x$x$x$x$x$x$x$x$x # 900
+
+# If setting this ``too high'' (TODO: yet to be determined), Emacs will crash
+# with a segmentation fault.
+n=20
+for i in $(seq 1 $n); do
+  # Roughly 2 KiB per message.  That is, we need two messages in order to
+  # exceed the typical size of the pipe buffer (4 KiB on commodity systems).
+  generate_message [subject]=$i-$x [from]=$i-$x@x.x
+done
+# With 20 messages à 2 KiB, we have about 10 full pipe buffers, which should be
+# enough to trigger the erroneous behavior.
+
+notmuch new > /dev/null
+
+test_begin_subtest 'Search for all messages'
+output=$(exec 2>&1; \
+         diff -wu \
+           <(notmuch search \* \
+               | sed \
+                   -e 's%^thread:[0-9a-f]*\ %%' \
+                   -e 's%;%%'; \
+             echo 'End of search results.'; \
+             echo) \
+           <(test_emacs 2>&1 \
+               '(notmuch-search "*") (notmuch-test-wait) (message (buffer-string))'))
+test_expect_equal "$output" ''
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index 9d77c0f..2f11eac 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -36,6 +36,7 @@ TESTS="
   encoding
   emacs
   maildir-sync
+  emacs-forgetfulness
 "
 
 # Clean up any results from a previous run
-- 
1.7.1

[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Thomas Schwinge]

This issue has been lying in ambush as of 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.

Signed-off-by: Thomas Schwinge <thomas@schwinge.name>
---
 emacs/notmuch.el |   70 +++++++++++++++++++++++++++++++----------------------
 1 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 3d82f0d..35ccee6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match."
     (propertize authors 'face 'notmuch-search-matching-authors)))
 
 (defun notmuch-search-insert-authors (format-string authors)
-  ;; Save the match data to avoid interfering with
-  ;; `notmuch-search-process-filter'.
-  (save-match-data
     (let* ((formatted-authors (format format-string authors))
 	   (formatted-sample (format format-string ""))
 	   (visible-string formatted-authors)
@@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match."
 	  (setq overlay (make-overlay start (point)))
 	  (overlay-put overlay 'invisible invis-spec)
 	  (overlay-put overlay 'isearch-open-invisible #'notmuch-search-isearch-authors-show)))
-      (insert padding))))
+      (insert padding)))
 
 (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
@@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors match."
 	  do (notmuch-search-insert-field field date count authors subject tags)))
   (insert "\n"))
 
+(defvar notmuch-search-process-filter-data nil
+  "Data that has not yet been processed.")
+(make-variable-buffer-local 'notmuch-search-process-filter-data)
+
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
   (let ((buffer (process-buffer proc))
@@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors match."
     (if (buffer-live-p buffer)
 	(with-current-buffer buffer
 	  (save-excursion
-	    (let ((line 0)
-		  (more t)
-		  (inhibit-read-only t))
-	      (while more
-		(if (string-match "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" string line)
-		    (let* ((thread-id (match-string 1 string))
-			   (date (match-string 2 string))
-			   (count (match-string 3 string))
-			   (authors (match-string 4 string))
-			   (subject (match-string 5 string))
-			   (tags (match-string 6 string))
-			   (tag-list (if tags (save-match-data (split-string tags)))))
-		      (goto-char (point-max))
-		      (let ((beg (point-marker)))
-			(notmuch-search-show-result date count authors subject tags)
-			(notmuch-search-color-line beg (point-marker) tag-list)
-			(put-text-property beg (point-marker) 'notmuch-search-thread-id thread-id)
-			(put-text-property beg (point-marker) 'notmuch-search-authors authors)
-			(put-text-property beg (point-marker) 'notmuch-search-subject subject)
-			(if (string= thread-id notmuch-search-target-thread)
-			    (progn
-			      (set 'found-target beg)
-			      (set 'notmuch-search-target-thread "found"))))
-		      (set 'line (match-end 0)))
-		  (set 'more nil)))))
+	    (let ((inhibit-read-only t)
+		  ;; We may have a partial line saved from the last iteration.
+		  (string (concat notmuch-search-process-filter-data string))
+		  (start 0))
+	      (goto-char (point-max))
+	      ;; Split `string' into lines.
+	      (while (string-match "\n" string start)
+		(let ((line (substring string start (match-beginning 0))))
+		  ;; Save the beginning of the next line already here, so that
+		  ;; we can mangle the match data later on.
+		  (setq start (match-end 0))
+		  (if (string-match
+		       "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$"
+		       line)
+		      (let* ((thread-id (match-string 1 line))
+			     (date (match-string 2 line))
+			     (count (match-string 3 line))
+			     (authors (match-string 4 line))
+			     (subject (match-string 5 line))
+			     (tags (match-string 6 line))
+			     (tag-list (if tags (split-string tags))))
+			(let ((beg (point-marker)))
+			  (notmuch-search-show-result date count authors subject tags)
+			  (notmuch-search-color-line beg (point-marker) tag-list)
+			  (put-text-property beg (point-marker) 'notmuch-search-thread-id thread-id)
+			  (put-text-property beg (point-marker) 'notmuch-search-authors authors)
+			  (put-text-property beg (point-marker) 'notmuch-search-subject subject)
+			  (if (string= thread-id notmuch-search-target-thread)
+			      (setq found-target beg
+				    notmuch-search-target-thread "found"))))
+		    ;; Non-conforming line.
+		    (insert (concat "Non-conforming line (ignored): <" line ">.\n")))))
+	      ;; Save the remainder after the last line break for the next
+	      ;; interation.
+	      (setq notmuch-search-process-filter-data (substring string start))))
 	  (if found-target
 	      (goto-char found-target)))
       (delete-process proc))))
@@ -858,6 +869,7 @@ The optional parameters are used as follows:
 		       "--sort=newest-first")
 		     query)))
 	  (set-process-sentinel proc 'notmuch-search-process-sentinel)
+	  (setq notmuch-search-process-filter-data nil)
 	  (set-process-filter proc 'notmuch-search-process-filter))))
     (run-hooks 'notmuch-search-hook)))
 
-- 
1.7.1

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Austin Clements]

Nice catch.

Is there a reason you keep the remaining data in a string instead of
taking the more idiomatic elisp approach of leaving it in the process
buffer?  In fact, the code would probably be simpler if you
immediately appended the string to the process buffer like a normal
process-filter and then peeled things away using buffer-oriented
regexp functions like looking-at.  Elisp is a lot better at
manipulating buffers than it is at manipulating strings.

On Wed, Feb 2, 2011 at 6:56 PM, Thomas Schwinge <thomas@schwinge.name> wrote:
> This issue has been lying in ambush as of 2009-11-24's commit
> 93af7b574598637c2766dd1f8ef343962c9a8efb.
>
> Signed-off-by: Thomas Schwinge <thomas@schwinge.name>
> ---
>  emacs/notmuch.el |   70 +++++++++++++++++++++++++++++++----------------------
>  1 files changed, 41 insertions(+), 29 deletions(-)
>
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 3d82f0d..35ccee6 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match."
>     (propertize authors 'face 'notmuch-search-matching-authors)))
>
>  (defun notmuch-search-insert-authors (format-string authors)
> -  ;; Save the match data to avoid interfering with
> -  ;; `notmuch-search-process-filter'.
> -  (save-match-data
>     (let* ((formatted-authors (format format-string authors))
>           (formatted-sample (format format-string ""))
>           (visible-string formatted-authors)
> @@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match."
>          (setq overlay (make-overlay start (point)))
>          (overlay-put overlay 'invisible invis-spec)
>          (overlay-put overlay 'isearch-open-invisible #'notmuch-search-isearch-authors-show)))
> -      (insert padding))))
> +      (insert padding)))
>
>  (defun notmuch-search-insert-field (field date count authors subject tags)
>   (cond
> @@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors match."
>          do (notmuch-search-insert-field field date count authors subject tags)))
>   (insert "\n"))
>
> +(defvar notmuch-search-process-filter-data nil
> +  "Data that has not yet been processed.")
> +(make-variable-buffer-local 'notmuch-search-process-filter-data)
> +
>  (defun notmuch-search-process-filter (proc string)
>   "Process and filter the output of \"notmuch search\""
>   (let ((buffer (process-buffer proc))
> @@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors match."
>     (if (buffer-live-p buffer)
>        (with-current-buffer buffer
>          (save-excursion
> -           (let ((line 0)
> -                 (more t)
> -                 (inhibit-read-only t))
> -             (while more
> -               (if (string-match "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" string line)
> -                   (let* ((thread-id (match-string 1 string))
> -                          (date (match-string 2 string))
> -                          (count (match-string 3 string))
> -                          (authors (match-string 4 string))
> -                          (subject (match-string 5 string))
> -                          (tags (match-string 6 string))
> -                          (tag-list (if tags (save-match-data (split-string tags)))))
> -                     (goto-char (point-max))
> -                     (let ((beg (point-marker)))
> -                       (notmuch-search-show-result date count authors subject tags)
> -                       (notmuch-search-color-line beg (point-marker) tag-list)
> -                       (put-text-property beg (point-marker) 'notmuch-search-thread-id thread-id)
> -                       (put-text-property beg (point-marker) 'notmuch-search-authors authors)
> -                       (put-text-property beg (point-marker) 'notmuch-search-subject subject)
> -                       (if (string= thread-id notmuch-search-target-thread)
> -                           (progn
> -                             (set 'found-target beg)
> -                             (set 'notmuch-search-target-thread "found"))))
> -                     (set 'line (match-end 0)))
> -                 (set 'more nil)))))
> +           (let ((inhibit-read-only t)
> +                 ;; We may have a partial line saved from the last iteration.
> +                 (string (concat notmuch-search-process-filter-data string))
> +                 (start 0))
> +             (goto-char (point-max))
> +             ;; Split `string' into lines.
> +             (while (string-match "\n" string start)
> +               (let ((line (substring string start (match-beginning 0))))
> +                 ;; Save the beginning of the next line already here, so that
> +                 ;; we can mangle the match data later on.
> +                 (setq start (match-end 0))
> +                 (if (string-match
> +                      "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$"
> +                      line)
> +                     (let* ((thread-id (match-string 1 line))
> +                            (date (match-string 2 line))
> +                            (count (match-string 3 line))
> +                            (authors (match-string 4 line))
> +                            (subject (match-string 5 line))
> +                            (tags (match-string 6 line))
> +                            (tag-list (if tags (split-string tags))))
> +                       (let ((beg (point-marker)))
> +                         (notmuch-search-show-result date count authors subject tags)
> +                         (notmuch-search-color-line beg (point-marker) tag-list)
> +                         (put-text-property beg (point-marker) 'notmuch-search-thread-id thread-id)
> +                         (put-text-property beg (point-marker) 'notmuch-search-authors authors)
> +                         (put-text-property beg (point-marker) 'notmuch-search-subject subject)
> +                         (if (string= thread-id notmuch-search-target-thread)
> +                             (setq found-target beg
> +                                   notmuch-search-target-thread "found"))))
> +                   ;; Non-conforming line.
> +                   (insert (concat "Non-conforming line (ignored): <" line ">.\n")))))
> +             ;; Save the remainder after the last line break for the next
> +             ;; interation.
> +             (setq notmuch-search-process-filter-data (substring string start))))
>          (if found-target
>              (goto-char found-target)))
>       (delete-process proc))))
> @@ -858,6 +869,7 @@ The optional parameters are used as follows:
>                       "--sort=newest-first")
>                     query)))
>          (set-process-sentinel proc 'notmuch-search-process-sentinel)
> +         (setq notmuch-search-process-filter-data nil)
>          (set-process-filter proc 'notmuch-search-process-filter))))
>     (run-hooks 'notmuch-search-hook)))
>
> --
> 1.7.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

Hallo!

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements <amdragon@mit.edu> wrote:
> Is there a reason you keep the remaining data in a string instead of
> taking the more idiomatic elisp approach of leaving it in the process
> buffer?  In fact, the code would probably be simpler if you
> immediately appended the string to the process buffer like a normal
> process-filter and then peeled things away using buffer-oriented
> regexp functions like looking-at.  Elisp is a lot better at
> manipulating buffers than it is at manipulating strings.

Ha, I hear you -- this is what I meant to do originally.  But then, the
save-in-string approach (even though I always considered keeping state in
the string a bit ugly) seemed more simple to me.  As I said: writing
elisp code is not my primary profession...  :-) (Perhaps I should buy a
book about it, or something.)  Now that you confirmed my original idea,
I'll see about re-writing the code accordingly, so thanks for the input!


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On Thu,  3 Feb 2011 00:56:39 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> This issue has been lying in ambush as of 2009-11-24's commit
> 93af7b574598637c2766dd1f8ef343962c9a8efb.

Thanks very much for tracking down this bug, Thomas. What a nasty bug to
have in notmuch!

Your fix seems to drop the last thread from the search results
view. I've now committed a slightly modified fix that avoids that
problem. I also made the test case provide slightly cleaner results.

Let me know if you see any problems.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements <amdragon@mit.edu> wrote:
> Nice catch.
> 
> Is there a reason you keep the remaining data in a string instead of
> taking the more idiomatic elisp approach of leaving it in the process
> buffer?

Thomas is excused since he was just modifying my code originally.

And now I've gone and made it even worse by adding a bunch of hideously
non-idiomatic expressions like:

	(while (and (< line (length string)) (= (elt string line) ?\n))
	  (setq line (1+ line)))

The rough equivalent in C would be quite natural (where we actually have
pointers):

	while (*s && *s == '\n')
            s++;

but the above elisp is quite nasty.

> In fact, the code would probably be simpler if you
> immediately appended the string to the process buffer like a normal
> process-filter and then peeled things away using buffer-oriented
> regexp functions like looking-at.  Elisp is a lot better at
> manipulating buffers than it is at manipulating strings.

I spent a bit of time trying this. Using looking-at is definitely better
than string-match since we can then use things like (match-string 1)
rather than (match-string 1 string). And getting rid of the "line"
variable means that all of the ugly expressions like the one I showed
above went away.

The approach I tried was to use a temporary buffer for the unparsed data
and the process buffer for the resulting parsed data. The code got a bit
awkward as I kept having to jump back and for between (with-temp-buffer)
and (with-current-buffer buffer)---particularly due to the buffer-local
variable to hold the trailing data from the past run.

A better approach might be to use a single buffer, leave the unparse
data at the end, and just make it hidden.

But I'll leave this alone for now since fighting elisp has left me
annoyed. If anyone wants to clean up my hideous elisp here, then that
would be fine with me.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 23495 bytes --]

Hi!

On Thu, 10 Mar 2011 18:02:09 -0800, Carl Worth <cworth@cworth.org> wrote:
> On Thu,  3 Feb 2011 00:56:39 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
> > This issue has been lying in ambush as of 2009-11-24's commit
> > 93af7b574598637c2766dd1f8ef343962c9a8efb.
> 
> Thanks very much for tracking down this bug, Thomas. What a nasty bug to
> have in notmuch!
> 
> Your fix seems to drop the last thread from the search results
> view. I've now committed a slightly modified fix that avoids that
> problem. I also made the test case provide slightly cleaner results.
> 
> Let me know if you see any problems.

That is much better, thanks!

But we're not there yet...  %-| That is, today I hit another issue that
appears to hide in the same elisp code.  See ``Error: Unexpected output
From notmuch search''.  (And, thanks to eBay for long subject lines...)

Unfortunately I'm totally out of time at the moment (final month of
writing and wrapping up my diploma thesis), so I'm just dumping my state
here, for now.

Screen scrape from Emacs UI:

     Today 09:47 [29/31] Francisco Cribari, Falcon1, Ian Ozsvald, Philip Aston, Clark Evans, Ed S, Koen, quarara, Malte Finsterwalder, Sukhikh Nick, Christian Prim, Eliah Kagan, nebula, Rik Theys, Seth Forshee, Nick Nguyen, aa-hcl, Launchpad Bug Tracker  [Bug 606238] Re: synaptic touchpad not recognized on dell latitude e6510 and others (direct inbox incoming unread)
     Today 02:10 [1/2]   David Bremner, Thomas Schwinge    [PATCH] dump: Don't sort. (direct inbox incoming list notmuch)
     Today 01:45 [81/110] bugzilla@redhat.com  [Bug 590880] Trackpad on Dell Inspiron E6410 not recognized (direct inbox incoming later unread waiting)
     Today 00:19 [2/10]  Bryce Harrington, Extender, juliobispo, Paul, karaluh, Eliah Kagan  [Bug 507062] Re: synaptic assert failure: synaptic: ../../src/xcb_io.c:385: _XAllocID: Assertion `ret != inval_id' failed. (direct inbox incoming unread)
     Yest. 23:01 [1/5]   Maren Jörlitschka, Julia Schmautz, Monika Humérez  Huhu (direct inbox incoming musical)
     Yest. 15:18 [1/1]   Birgit auf Reisen    [Neuer Artikel] Willkommen zurück :) (direct inbox incoming)
     Yest. 13:29 [1/1]   Matthias Bauch       Ein Fest... (direct inbox incoming)
     Yest. 07:09 [1/3]   Quantumsolutions UG, Thomas Schwinge  Rechnung für Artikel Mustang 4GB DDR2 800MHz Ram Speicher Pc6400 Pc Dimm 800( Artikelnummer: 350365327282 )... (attachment debit direct ebay inbox incoming replied signed)
      Sun. 22:56 [2/2]   Jan Harnisch, Denis Hückel     Proben... (direct inbox incoming nachtspaziergang)
      Sun. 22:16 [1/1]   Maksym Planeta       Fixing tmpfs and defpager (attachment direct hurd inbox incoming list)
      Sun. 18:41 [1/1]   eBay                 Ihre Rechnung für eBay-Käufe: Motörhead : The Best of -- 2 CD`s -- (170720368449#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Motörhead : Snake Bite Love (170724295430#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),METAL BALLADS : Vol . 2 (170724323789#),Brother Cane : Same (170720367821#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),Def Leppard : Adrenalize (170720367895#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Ozzy Osbourne : No more Tears (170720395234#),Nightwish : Wishmaster (170720426915#),Motörhead : No sleep`til Hammersmith (170720368358#),Def Leppard : High`n`Dry (170720367981#),The Moody Blues : Days of Future Past (170724296562#),JACKYL : Same (170720368246#),Judas Priest : Stained Class (170720394999#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)
      Sun. 18:35 [1/1]   eBay                 Herzlichen Glückwunsch, der Artikel METAL BALLADS : Vol . 2 gehört Ihnen! (direct ebay inbox incoming unread)
      Sun. 18:02 [1/1]   eBay                 Ihre Rechnung für eBay-Käufe: Motörhead : The Best of -- 2 CD`s -- (170720368449#),Ozzy Osbourne : No more Tears (170720395234#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Nightwish : Wishmaster (170720426915#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Motörhead : Snake Bite Love (170724295430#),Motörhead : No sleep`til Hammersmith (170720368358#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),Brother Cane : Same (170720367821#),Def Leppard : High`n`Dry (170720367981#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),The Moody Blues : Days of Future Past (170724296562#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),JACKYL : Same (170720368246#),Def Leppard : Adrenalize (170720367895#)
    Error: Unexpected output from notmuch search:
    ,Judas Priest : Stained Class (170720394999#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)
    
      Sun. 17:59 [1/1]   eBay                 Herzlichen Glückwunsch, der Artikel Wolfhound / Anne Haigis : Truth -- Jeton -- gehört Ihnen! (direct ebay inbox incoming unread)
      Sun. 17:59 [1/1]   eBay                 Herzlichen Glückwunsch, der Artikel The Moody Blues : Days of Future Past gehört Ihnen! (direct ebay inbox incoming unread)
    [...]

Part of strace of notmuch:

    [...]
    write(1, "thread:0000000000002889  Today 09:47 [29/31] Francisco Cribari, Falcon1, Ian Ozsvald, Philip Aston, Clark Evans, Ed S, Koen, quarara, Malte Finsterwalder, Sukhikh Nick, Christian Prim, Eliah Kagan, nebula, Rik Theys, Seth Forshee, Nick Nguyen| aa-hcl, Launchpad Bug Tracker; [Bug 606238] Re: synaptic touchpad not recognized on dell latitude e6510 and others (direct inbox incoming unread)\nthread:000000000002d069  Today 02:10 [1/2] David Bremner| Thomas Schwinge; [PATCH] dump: Don't sort. (direct inbox incoming list notmuch)\nthread:0000000000001980  Today 01:45 [81/110] bugzilla@redhat.com; [Bug 590880] Trackpad on Dell Inspiron E6410 not recognized (direct inbox incoming later unread waiting)\nthread:0000000000008628  Today 00:19 [2/10] Bryce Harrington| Extender, juliobispo, Paul, karaluh, Eliah Kagan; [Bug 507062] Re: synaptic assert failure: synaptic: ../../src/xcb_io.c:385: _XAllocID: Assertion `ret != inval_id' failed. (direct inbox incoming unread)\nthread:0000000000034f79  Yest. 23:01 [1/5] Maren J\303\266rlitschka| Julia Schmautz, Monika Hum\303\251rez; Huhu (direct inbox incoming musical)\nthread:0000000000035c8c  Yest. 15:18 [1/1] Birgit aufReisen; [Neuer Artikel] Willkommen zur\303\274ck :) (direct inbox incoming)\nthread:0000000000035c66  Yest. 13:29 [1/1] Matthias Bauch; Ein Fest... (direct inbox incoming)\nthread:00000000000309c2  Yest. 07:09 [1/3] Quantumsolutions UG| Thomas Schwinge; Rechnung f\303\274r Artikel Mustang 4GB DDR2 800MHz Ram Speicher Pc6400 Pc Dimm 800( Artikelnummer: 350365327282 )... (attachment debit direct ebay inbox incoming replied signed)\nthread:0000000000035b63   Sun. 22:56 [2/2] Jan Harnisch, Denis H\303\274ckel; Proben... (direct inbox incoming nachtspaziergang)\nthread:0000000000035b7a   Sun. 22:16 [1/1] Maksym Planeta; Fixing tmpfs and defpager (attachment direct hurd inbox incoming list)\nthread:0000000000035b5b   Sun. 18:41 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Mot\303\266rhead : Snake Bite Love (170724295430#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),METAL BALLADS : Vol . 2 (170724323789#),Brother Cane : Same (170720367821#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),Def Leppard : Adrenalize (170720367895#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Ozzy Osbourne : No more Tears (170720395234#),Nightwish : Wishmaster (170720426915#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Def Leppard : High`n`Dry (170720367981#),The Moody Blues : Days of Future Past (170724296562#),JACKYL : Same (170720368246#),Judas Priest : Stained Class (170720394999#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b5a   Sun. 18:35 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel METAL BALLADS : Vol . 2 geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b46   Sun. 18:02 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Ozzy Osbourne : No more Tears (170720395234#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Nightwish : Wishmaster (170720426915#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Mot\303\266rhead : Snake Bite Love (170724295430#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),Brother Cane : Same (170720367821#),Def Leppard : High`n`Dry (170720367981#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987(170720427089#),The Moody Blues : Days of Future Past (170724296562#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),JACKYL : Same (170720368246#),Def Leppard : Adrenalize (170720367895#)", 4096) = 4096
    [...]
    write(1, ",Judas Priest : Stained Class (170720394999#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b41   Sun. 17:59 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Wolfhound / Anne Haigis : Truth -- Jeton -- geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b40   Sun. 17:59 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel The Moody Blues : Days of Future Past geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b3f   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Ozzy Osbourne : No Rest for the Wicked geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b42   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Mot\303\266rhead : Snake Bite Love geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b3e   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Black Sabbath : Sabbath Bloody Sabbath geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b2d   Sun. 17:12 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Ozzy Osbourne : No more Tears (170720395234#),Nightwish : Wishmaster (170720426915#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Goethes Erben : Die Brut (170724254795#),Brother Cane : Same (170720367821#),Def Leppard : High`n`Dry (170720367981#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),Kiss : Revenge (170720395093#),GIANT : Time to Burn (170720394773#),JACKYL : Same (170720368246#),Def Leppard : Adrenalize (170720367895#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Judas Priest : Stained Class (170720394999#),DIO : Dream Evil (170724254266#),Rollins Band : Come in and Burn (170720395418#),God Forbid : Gone Forever (170720426298#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b28   Sun. 17:10 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Goethes Erben : Die Brut geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b25   Sun. 17:10 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Goethes Erben : Der Traum an die Erinnerung geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b26   Sun. 17:09 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel DIO : Dream Evil geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b00   Sun. 08:06 [1/1] Dyn; New uses for Dyn products, holiday email delivery, ad platforms and DNS (direct inbox incoming unread)\nthread:0000000000034f7e   Sat. 21:49 [9/15] Ludovic Court\303\250s, Samuel Thibault, Guillem Jover, Svante Signell| Thomas Schwinge; [PATCH gnumach] Inline version.m4 into configure.ac (attachment direct hurd inbox list replied signed)\nthread:000000000000060c   Sat. 16:31 [2/2] jhansonxi, Cameron Smith; [Bug 226823] Re: Use system proxy settings doesn't take care of proxy exceptions list in Gnome proxy settings (direct inbox)\nthread:000000000003587b   Sat. 09:57 [1/1] regioactive.de News; ZEHN: Interview mit Jubil\303\244umsfestival-Headliner Bodi Bill. Au\303\237erdem: Annett Louisan, Lenny Kravitz u.v.m. (direct inbox unread)\nthread:0000000000035875   Fri. 23:58 [1/1] HANSEPLATTE SHOP; Der Sackbahnhof im Plakatdesign: Poster von Rocket & Wink (direct inbox unread)\nthread:000000000000043d   Fri. 22:20 [3/9] razor, Curtis Hovey| Costales, Martin Pitt, Teej, Vlad, trevi; [Bug 35223] Re: Laptop TFT monitor - brightness level is not saved (direct inbox later unread)\nthread:00000000000019e5   Fri. 22:09 [43/52] sillyxone, Fran\303\247ois Isabelle, Ladislav Nesnera, Jim Ramsay, Raghu, cement_head, Mihai Secasiu, Brad Heap, bricedebrignaisplage, David Cramer, Ubuntu QA's Bug Bot, Robert, Andr\303\251, Sam Dodrill, Stan Schymanski, Clark Evans, Ed S, Koen, Donatas Burba, quarara, Rik Theys| sergiomb, MikeE, Nacho Coloma, Fabi, Duleep, Gaurav Padia; [Bug 7", 4096) = 4096
    [...]

Up to here, notmuch has written to stdout two 4096 bytes slices.  This
looks correct.

Part of strace on Emacs:

    [...]
    read(17, "thread:0000000000002889  Today 09:47 [29/31] Francisco Cribari, Falcon1, Ian Ozsvald, Philip Aston, Clark Evans, Ed S, Koen, quarara, Malte Finsterwalder, Sukhikh Nick, Christian Prim, Eliah Kagan, nebula, Rik Theys, Seth Forshee, Nick Nguyen| aa-hcl, Launchpad Bug Tracker; [Bug 606238] Re: synaptic touchpad not recognized on dell latitude e6510 and others (direct inbox incoming unread)\nthread:000000000002d069  Today 02:10 [1/2] David Bremner| Thomas Schwinge; [PATCH] dump: Don't sort. (direct inbox incoming list notmuch)\nthread:0000000000001980  Today 01:45 [81/110] bugzilla@redhat.com; [Bug 590880] Trackpad on Dell Inspiron E6410 not recognized (direct inbox incoming later unread waiting)\nthread:0000000000008628  Today 00:19 [2/10] Bryce Harrington| Extender, juliobispo, Paul, karaluh, Eliah Kagan; [Bug 507062] Re: synaptic assert failure: synaptic: ../../src/xcb_io.c:385: _XAllocID: Assertion `ret != inval_id' failed. (direct inbox incoming unread)\nthread:0000000000034f79  Yest. 23:01 [1/5] Maren J\303\266rlitschka| Julia Schmautz, Monika Hum\303\251rez; Huhu (direct inbox incoming musical)\nthread:0000000000035c8c  Yest. 15:18 [1/1] Birgit auf Reisen; [Neuer Artikel] Willkommen zur\303\274ck :) (direct inbox incoming)\nthread:0000000000035c66  Yest. 13:29 [1/1] Matthias Bauch; Ein Fest... (direct inbox incoming)\nthread:00000000000309c2  Yest. 07:09 [1/3] Quantumsolutions UG| Thomas Schwinge; Rechnung f\303\274r Artikel Mustang 4GB DDR2 800MHz Ram Speicher Pc6400 Pc Dimm 800( Artikelnummer: 350365327282 )... (attachment debit direct ebay inbox incoming replied signed)\nthread:0000000000035b63   Sun. 22:56 [2/2] Jan Harnisch, Denis H\303\274ckel; Proben... (direct inbox incoming nachtspaziergang)\nthread:0000000000035b7a   Sun. 22:16 [1/1] Maksym Planeta; Fixing tmpfs and defpager (attachment direct hurd inbox incoming list)\nthread:0000000000035b5b   Sun. 18:41 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Mot\303\266rhead : Snake Bite Love (170724295430#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),METAL BALLADS : Vol . 2 (170724323789#),Brother Cane : Same (170720367821#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),Def Leppard : Adrenalize (170720367895#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Ozzy Osbourne : No more Tears (170720395234#),Nightwish : Wishmaster (170720426915#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Def Leppard : High`n`Dry (170720367981#),The Moody Blues : Days of Future Past (170724296562#),JACKYL : Same (170720368246#),Judas Priest : Stained Class (170720394999#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b5a   Sun. 18:35 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel METAL BALLADS : Vol . 2 geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b46   Sun. 18:02 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Ozzy Osbourne : No more Tears (170720395234#),Wolfhound / Anne Haigis : Truth -- Jeton -- (170724297186#),Nightwish : Wishmaster (170720426915#),Black Sabbath : Sabbath Bloody Sabbath (170724295045#),Mot\303\266rhead : Snake Bite Love (170724295430#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Goethes Erben : Die Brut (170724254795#),Ozzy Osbourne : No Rest for the Wicked (170724295884#),Brother Cane : Same (170720367821#),Def Leppard : High`n`Dry (170720367981#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),The Moody Blues : Days of Future Past (170724296562#),GIANT : Time to Burn (170720394773#),Kiss : Revenge (170720395093#),JACKYL : Same (170720368246#),Def Leppard : Adrenalize (170720367895#", 4096) = 4095
    [...]
    read(17, ")", 4096)                     = 1
    [...]
    read(17, ",Judas Priest : Stained Class (170720394999#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Rollins Band : Come in and Burn (170720395418#),God Forbid : Gone Forever (170720426298#),DIO : Dream Evil (170724254266#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b41   Sun. 17:59 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Wolfhound / Anne Haigis : Truth -- Jeton -- geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b40   Sun. 17:59 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel The Moody Blues : Days of Future Past geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b3f   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Ozzy Osbourne : No Rest for the Wicked geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b42   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Mot\303\266rhead : Snake Bite Love geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b3e   Sun. 17:58 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Black Sabbath : Sabbath Bloody Sabbath geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b2d   Sun. 17:12 [1/1] eBay; Ihre Rechnung f\303\274r eBay-K\303\244ufe: Mot\303\266rhead : The Best of -- 2 CD`s -- (170720368449#),Ozzy Osbourne : No more Tears (170720395234#),Nightwish : Wishmaster (170720426915#),Mot\303\266rhead : No sleep`til Hammersmith (170720368358#),Goethes Erben : Die Brut (170724254795#),Brother Cane : Same (170720367821#),Def Leppard : High`n`Dry (170720367981#),In Extremo : Mein rasend Herz (170720368143#),Whitesnake : 1987 (170720427089#),Kiss : Revenge (170720395093#),GIANT : Time to Burn (170720394773#),JACKYL : Same (170720368246#),Def Leppard : Adrenalize (170720367895#),Goethes Erben : Der Traum an die Erinnerung (170724254679#),Judas Priest : Stained Class (170720394999#),DIO : Dream Evil (170724254266#),Rollins Band : Come in and Burn (170720395418#),God Forbid : Gone Forever (170720426298#),Don Dokken : Up from the Ashes (170720368061#) (direct ebay inbox incoming unread)\nthread:0000000000035b28   Sun. 17:10 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Goethes Erben : Die Brut geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b25   Sun. 17:10 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel Goethes Erben : Der Traum an die Erinnerung geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b26   Sun. 17:09 [1/1] eBay; Herzlichen Gl\303\274ckwunsch, der Artikel DIO : Dream Evil geh\303\266rt Ihnen! (direct ebay inbox incoming unread)\nthread:0000000000035b00   Sun. 08:06 [1/1] Dyn; New uses for Dyn products, holiday email delivery, ad platforms and DNS (direct inbox incoming unread)\nthread:0000000000034f7e   Sat. 21:49 [9/15] Ludovic Court\303\250s, Samuel Thibault, Guillem Jover, Svante Signell| Thomas Schwinge; [PATCH gnumach] Inline version.m4 into configure.ac (attachment direct hurd inbox list replied signed)\nthread:000000000000060c   Sat. 16:31 [2/2] jhansonxi, Cameron Smith; [Bug 226823] Re: Use system proxy settings doesn't take care of proxy exceptions list in Gnome proxy settings (direct inbox)\nthread:000000000003587b   Sat. 09:57 [1/1] regioactive.de News; ZEHN: Interview mit Jubil\303\244umsfestival-Headliner Bodi Bill. Au\303\237erdem: Annett Louisan, Lenny Kravitz u.v.m. (direct inbox unread)\nthread:0000000000035875   Fri. 23:58 [1/1] HANSEPLATTE SHOP; Der Sackbahnhof im Plakatdesign: Poster von Rocket & Wink (direct inbox unread)\nthread:000000000000043d   Fri. 22:20 [3/9] razor, Curtis Hovey| Costales, Martin Pitt, Teej, Vlad, trevi; [Bug 35223] Re: Laptop TFT monitor - brightness level is not saved (direct inbox later unread)\nthread:00000000000019e5   Fri. 22:09 [43/52] sillyxone, Fran\303\247ois Isabelle, Ladislav Nesnera, Jim Ramsay, Raghu, cement_head, Mihai Secasiu, Brad Heap, bricedebrignaisplage, David Cramer, Ubuntu QA's Bug Bot, Robert, Andr\303\251, Sam Dodrill, Stan Schymanski, Clark Evans, Ed S, Koen, Donatas Burba, quarara, Rik Theys| sergiomb, MikeE, Nacho Coloma, Fabi, Duleep, Gaurav Padia; [Bug ", 4096) = 4095
    [...]
    read(17, "7", 4096)                     = 1
    [...]

Up to here, Emacs has read slices of 4095, 1, 4095, 1 bytes.  (I've
noticed and reported this before, and still wonder why read would only
return 4095 bytes if asked for 4096, and these should be available.
Kernel issue?)  Yet, this also looks correct.

Thus, I'd suspect the elisp code again.


Grüße,
 Thomas


[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

[Austin Clements]

On Tue, 15 Nov 2011, Thomas Schwinge <thomas@schwinge.name> wrote:
> Hi!
>
> On Thu, 10 Mar 2011 18:02:09 -0800, Carl Worth <cworth@cworth.org> wrote:
>> On Thu,  3 Feb 2011 00:56:39 +0100, Thomas Schwinge <thomas@schwinge.name> wrote:
>> > This issue has been lying in ambush as of 2009-11-24's commit
>> > 93af7b574598637c2766dd1f8ef343962c9a8efb.
>> 
>> Thanks very much for tracking down this bug, Thomas. What a nasty bug to
>> have in notmuch!
>> 
>> Your fix seems to drop the last thread from the search results
>> view. I've now committed a slightly modified fix that avoids that
>> problem. I also made the test case provide slightly cleaner results.
>> 
>> Let me know if you see any problems.
>
> That is much better, thanks!
>
> But we're not there yet...  %-| That is, today I hit another issue that
> appears to hide in the same elisp code.  See ``Error: Unexpected output
> From notmuch search''.  (And, thanks to eBay for long subject lines...)
>
> Unfortunately I'm totally out of time at the moment (final month of
> writing and wrapping up my diploma thesis), so I'm just dumping my state
> here, for now.

*snip*

I happened to notice that this is still marked as a bug.  I'm marking it
as fixed.  This code was completely rewritten to use the streaming JSON
parser (and now the streaming S-expression parser) and there's now a
test that feeds the process filter the search data one character at a
time to catch these sorts of buffer boundary bugs.