undigestify is fooled by messages containing lines of dashes

undigestify is fooled by messages containing lines of dashes

[Tim Hesterberg]

[-- Attachment #1: Type: text/plain, Size: 5352 bytes --]

Here is 
* a review of a problem I reported previously, 
* a new problem that occurs with the workaround.

On 10 Aug 2007 I reported:
> Using emacs 21.3, undigestify-rmail-message works fine, giving a large
> number of individual messages.
>
> Using emacs 22.1, the result is only two message:
> * correct first message, the "Today's Topics" message
> * all other real messages are kept as part of a single message.
>
> I get the same result using --no-init-file.
> ...
>
> If I use emacs 22.1 but load the emacs 21.3 version of
> .../lisp/mail/undigest.el, then undigestify works correctly.
Since then I've been using the version of undigest.el from emacs 21.3.

There is a new problem that occurs with this solution - that undigestify
thinks a line of dashes
--------------------------------------------------
separates messages, hence splits messages containing such
lines into multiple messages.

I'll attach an rmail file containing a single message, a digest
message from a mailing list.  Message 14 in the digest is a relatively
long message, containing multiple topics separated by dashed lines.

To reproduce this, load undigest.el from emacs 21.3, 
open the file using rmail, and undigestify the digest.  That long
message 14 is split into multiple messages, with all but the first having
a blank subject.  Here is the rmail summary:

    1D  6-Feb  -devel-request@r-project.[1335] R-devel Digest, Vol 60, Issue 6
    2   6-Feb  -devel-request@r-project.  [43] R-devel Digest, Vol 60, Issue 6
    3-  4-Feb     scott.yonker@gmail.com  [24] [Rd] garchFit (PR#10698)
...
   15-  6-Feb   p.murrell@auckland.ac.nz [141] Re: [Rd] Error in grid.text after tcltk package loaded
   16-  6-Feb  to: r-devel@stat.math.eth  [10] [Rd] Suggestions for R-intro manual (PR#10701)
   17-         to: r-devel@r-project.org  [23] 
   18-         to: r-devel@r-project.org  [18] 
   19-         to: r-devel@r-project.org  [10] 
   20-         to: r-devel@r-project.org  [14] 
   21-         to: r-devel@r-project.org   [8] 
   22-         to: r-devel@r-project.org  [14] 
   23-         to: r-devel@r-project.org   [7] 
   24-         to: r-devel@r-project.org   [9] 
   25-         to: r-devel@r-project.org  [31] 
   26-  6-Feb      ripley@stats.ox.ac.uk  [47] Re: [Rd] R valgrind question
   27          to: r-devel@r-project.org   [8] 

In contrast, the list of topics within the original digest:

Today's Topics:

   1. garchFit (PR#10698) (scott.yonker@gmail.com)
...
  13. Re: Error in grid.text after tcltk package loaded (Paul Murrell)
  14. Suggestions for R-intro manual (PR#10701) (timh@insightful.com)
  15. Re: R valgrind question (Prof Brian Ripley)

If I start emacs with --no-init-file I can reproduce both bugs;
failure to split into component messages with the current version of
undigest.el, and extra splits using the old version.




In GNU Emacs 22.1.1 (i386-mingw-nt5.1.2600)
 of 2007-06-02 on RELEASE
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4) --cflags -Ic:/gnuwin32/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  locale-coding-system: cp1252
  default-enable-multibyte-characters: t

Major mode: RMAIL Summary

Minor modes in effect:
  display-time-mode: t
  shell-dirtrack-mode: t
  show-paren-mode: t
  encoded-kbd-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
SPC a n d SPC e x t r a SPC s p l i t s SPC u s i n 
g SPC t h e SPC o l d SPC v e r s i o n . C-n C-SPC 
C-n C-n C-n C-n C-n C-w C-u C-x r t e m p <return> 
C-x k <return> C-x 1 <escape> v C-u C-u C-p C-u C-p 
C-u C-p C-u C-p C-e a SPC l i n e C-n C-n C-u C-f C-u 
C-f C-u C-f C-f <escape> d s p l i t s C-n C-a <escape> 
d C-d C-u C-p C-n C-e SPC o f SPC d a s y e s <backspace> 
<backspace> <backspace> h e s C-u C-p C-u C-p C-u C-p 
C-u C-p C-p C-p C-p C-u C-f C-u C-f C-e <backspace> 
SPC <backspace> . C-p C-p C-p C-p C-e <backspace> <backspace> 
<backspace> <backspace> <backspace> C-n C-n C-n C-a 
C-SPC <escape> > C-x C-x C-SPC C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n <escape> w C-x 
N C-x b t e m p 2 <return> C-y <escape> < C-x N C-x 
k <return> C-u C-x r t e m p <return> <escape> x u 
n d i g e s t i f y <return> h <escape> x e m a c s 
- r e p <tab> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
r e p o <tab> r t <tab> <return>

Recent messages:
Counting messages...done
Mark set [3 times]
Saved text from "First some background on a problem I rep"
Mark set [2 times]
Counting messages...done
Message successfully undigestified
Counting messages...done
Computing summary lines...done
Making completion list...
Loading emacsbug...done

[-- Attachment #2: temp --]
[-- Type: application/octet-stream, Size: 48634 bytes --]

Re: undigestify is fooled by messages containing lines of dashes

[Glenn Morris]

Tim Hesterberg wrote:

> Since then I've been using the version of undigest.el from emacs 21.3.

That kind of thing is generally inadvisable.

> There is a new problem that occurs with this solution - that undigestify
> thinks a line of dashes
> --------------------------------------------------
> separates messages, hence splits messages containing such
> lines into multiple messages.

Please try the Emacs-22.1.90 pretest available from
ftp://alpha.gnu.org/gnu/emacs/pretest, where this issue should be fixed.

Re: undigestify is fooled by messages containing lines of dashes

[Tim Hesterberg]

Thank you for the response.

Unfortunately, I am unable to get mail from the POP server
when using 22.1.90 pretest.  I reported that separately to
emacs-pretest-bug@gnu.org .

>Tim Hesterberg wrote:
>
>> Since then I've been using the version of undigest.el from emacs 21.3.
>
>That kind of thing is generally inadvisable.
>
>> There is a new problem that occurs with this solution - that undigestify
>> thinks a line of dashes
>> --------------------------------------------------
>> separates messages, hence splits messages containing such
>> lines into multiple messages.
>
>Please try the Emacs-22.1.90 pretest available from
>ftp://alpha.gnu.org/gnu/emacs/pretest, where this issue should be fixed.

Re: undigestify is fooled by messages containing lines of dashes

[Glenn Morris]

Tim Hesterberg wrote:

> Unfortunately, I am unable to get mail from the POP server when
> using 22.1.90 pretest.

Thanks for trying it. Sorry about the new problem. undigest does seem
to work for me on the example you attached.

>  I reported that separately to emacs-pretest-bug@gnu.org .

I haven't seen that mail arrive yet (they get forwarded to
emacs-devel@gnu.org these days), but perhaps it is held up due to
moderation or something.

Re: undigestify is fooled by messages containing lines of dashes

[Glenn Morris]

Glenn Morris wrote:

> Tim Hesterberg wrote:
>
>> Unfortunately, I am unable to get mail from the POP server when
>> using 22.1.90 pretest.
[...]
>>  I reported that separately to emacs-pretest-bug@gnu.org .
>
> I haven't seen that mail arrive yet

Still haven't seen that mail. Can you post the problem here?

Re: undigestify is fooled by messages containing lines of dashes

[Tim Hesterberg]

>Glenn Morris wrote:
>
>> Tim Hesterberg wrote:
>>
>>> Unfortunately, I am unable to get mail from the POP server when
>>> using 22.1.90 pretest.
>[...]
>>>  I reported that separately to emacs-pretest-bug@gnu.org .
>>
>> I haven't seen that mail arrive yet
>
>Still haven't seen that mail. Can you post the problem here?

I'm trying 22.1.90, and for comparison 22.1.1.

I start emacs using -no-init-file, then run
(prog1
  (setenv "MAILHOST" "sewinexch00.insightful.com")  ; 7/10/06
  (setq rmail-primary-inbox-list '("po:timh") 
  	rmail-pop-password-required t)
)
before M-x rmail
On both systems it asks for my POP Password, and on 22.1.1 proceeds
to read the mail from the server into my RMAIL file.  On 22.1.90
I get a message
	Wrong type argument: stringp, nil

The *Messages* buffer contains:
Loading rmail...done
Counting messages...done
Getting mail from the remote server ...
apply: Wrong type argument: stringp, nil


If I try M-x rmail again, I get the same message.

From the message I'd guess that a password is being set as nil instead
of string.
rmail-encoded-remote-password is a string
rmail-pop-password is nil
rmail-remote-password is nil


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
d:/emacs/emacs-22.1.90/etc/DEBUG for instructions.


In GNU Emacs 22.1.90.1 (i386-mingw-nt5.1.2600)
 of 2008-01-30 on PRETEST
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4) --cflags -Ic:/gnuwin32/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  locale-coding-system: cp1252
  default-enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  shell-dirtrack-mode: t
  show-paren-mode: t
  encoded-kbd-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
<return> <escape> x r e p o r <tab> <return>

Recent messages:
Loading comint...done
Loading tex-mode...done
Loading tex-mode-rmh...done
For information about GNU Emacs and the GNU system, type C-h C-a.
"sewinexch00.insightful.com"
Loading rmail...done
Counting messages...done
Getting mail from the remote server ...
apply: Wrong type argument: stringp, nil
Loading emacsbug...done

rmail pop error [was Re: undigestify is fooled by messages containing lines of dashes]

[Glenn Morris]

Tim Hesterberg wrote:

> I start emacs using -no-init-file, then run
> (prog1
>   (setenv "MAILHOST" "sewinexch00.insightful.com")  ; 7/10/06
>   (setq rmail-primary-inbox-list '("po:timh") 
>   	rmail-pop-password-required t)
> )
> before M-x rmail
> On both systems it asks for my POP Password, and on 22.1.1 proceeds
> to read the mail from the server into my RMAIL file.  On 22.1.90
> I get a message
> 	Wrong type argument: stringp, nil
>
> The *Messages* buffer contains:
> Loading rmail...done
> Counting messages...done
> Getting mail from the remote server ...
> apply: Wrong type argument: stringp, nil


Please do M-x toggle-debug-on-error, provoke the error again, and post
the backtrace.

What are the values of rmail-movemail-program and
rmail-movemail-variant-in-use after you get the error?

What does (rmail-autodetect) return?

Is there a `movemail' executable in exec-directory or exec-path, and
do you have permission to run it? What does `movemail --version' output?

Re: rmail pop error

[Glenn Morris]

Glenn Morris wrote:

> Please do M-x toggle-debug-on-error, provoke the error again, and post
> the backtrace.
>
> What are the values of rmail-movemail-program and
> rmail-movemail-variant-in-use after you get the error?
>
> What does (rmail-autodetect) return?
>
> Is there a `movemail' executable in exec-directory or exec-path, and
> do you have permission to run it? What does `movemail --version' output?


It looks to me like rmail-movemail-program is unset for you for some
reason, but I'm afraid I can't debug this without more information,
since it works for me.

Re: rmail pop error

[Tim Hesterberg]

>Glenn Morris wrote:
>
>> Please do M-x toggle-debug-on-error, provoke the error again, and post
>> the backtrace.
>>
>> What are the values of rmail-movemail-program and
>> rmail-movemail-variant-in-use after you get the error?
>>
>> What does (rmail-autodetect) return?
>>
>> Is there a `movemail' executable in exec-directory or exec-path, and
>> do you have permission to run it? What does `movemail --version' output?
>
>It looks to me like rmail-movemail-program is unset for you for some
>reason, but I'm afraid I can't debug this without more information,
>since it works for me.

Sorry, I didn't see the first message when it came out.  Answers below.


As before, I start with -no-init-file, then do:
(prog1
  (setenv "MAILHOST" "sewinexch00.insightful.com")  ; 7/10/06
  (setq rmail-primary-inbox-list '("po:timh") 
  	rmail-pop-password-required t)
)
(Then 'M-x rmail' works fine in emacs 22.1.1)

In emacs 22.1.90, I now do 'M-x toggle-debug-on-error' and 'M-x rmail'.
The backtrace is:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  call-process(nil nil #<buffer  *temp*> nil "po:timh" "h:/.newmail-timh" "MYPASSWORDomittedHERE")
  apply(call-process (nil nil #<buffer  *temp*> nil "po:timh" "h:/.newmail-timh" "MYPASSWORDomittedHERE"))
  rmail-insert-inbox-text(("po:timh") t)
  rmail-get-new-mail()
  rmail()
  call-interactively(rmail)
  execute-extended-command(nil)
  call-interactively(execute-extended-command)

The values of the two variables are:
rmail-movemail-program is nil
rmail-movemail-variant-in-use is nil

This is the same as in emacs 22.1.1
They were both unset before calling M-x rmail.

--------------------------------------------------
(rmail-autodetect)
returns nil in both 22.1.90 and 22.1.1  (after M-x rmail)

--------------------------------------------------
exec-directory is a variable defined in `C source code'.
Its value is 
"d:/emacs/emacs-22.1.90/bin/"

exec-path is a variable defined in `C source code'.
Its value is 
("d:/cygwin/bin" "d:/emacs/gnuserv" "d:/texmf/miktex/bin" "c:/Rtools/perl/bin" "c:/Rtools/MinGW/bin" "C:/Perl/bin" "D:/local/MiKTeX2.6/miktex/bin" "C:/Program Files/Insightful/splus80" "C:/Program Files/Insightful/splus80/cmd" "C:/WINDOWS/system32" "C:/WINDOWS" "C:/WINDOWS/System32/Wbem" "C:/Program Files/Intel/DMIX" "C:/Program Files/Perforce" "C:/Progra~1/R/R-2.6.1/bin" "c:/Program Files/HTML Help Workshop" "." "d:/cygwin/usr/local/bin" "d:/cygwin/bin" "d:/emacs/gnuserv" "d:/emacs/emacs-22.1.90/bin")

This does exist, and is executable
d:/emacs/emacs-22.1.90/bin/movemail.exe

There is no other movemail program in the other directories in exec-path.

--------------------------------------------------
On a lark, I tried:

(setq rmail-movemail-variant-in-use "emacs")

Then M-x rmail gives a different trace:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  call-process(nil nil #<buffer  *temp*> nil "po:timh" "h:/.newmail-timh" "MYPASSWORDomittedHERE")
  apply(call-process (nil nil #<buffer  *temp*> nil "po:timh" "h:/.newmail-timh" "MYPASSWORDomittedHERE"))
  rmail-insert-inbox-text(("po:timh") t)
  rmail-get-new-mail()
  rmail()
  call-interactively(rmail)
  execute-extended-command(nil)
  call-interactively(execute-extended-command)
  recursive-edit()
  byte-code("Æ\x10	@Ç=ƒ!

Re: rmail pop error

[Glenn Morris]

Tim Hesterberg wrote:

> (rmail-autodetect)
> returns nil in both 22.1.90 and 22.1.1  (after M-x rmail)
[...]
> This does exist, and is executable
> d:/emacs/emacs-22.1.90/bin/movemail.exe

Thanks; I think I see what's going on now. Does it work in 22.1.90 if
you add:

(setq rmail-movemail-program "d:/emacs/emacs-22.1.90/bin/movemail.exe")

to your setup?

Without doing that, does this patch fix it? (Apply to rmail.el in
22.1.90, recompile the file, restart Emacs.)


*** rmail.el	31 Jan 2008 10:39:19 -0000	1.433.2.9
--- rmail.el	20 Feb 2008 02:11:19 -0000
***************
*** 214,220 ****
        (dolist (dir (append rmail-movemail-search-path exec-path
  			   (list exec-directory)))
  	(when (and dir (file-accessible-directory-p dir))
! 	  (let ((progname (expand-file-name "movemail" dir)))
  	    (when (and (not (file-directory-p progname))
  		       (file-executable-p progname))
  	      (let ((x (rmail-probe progname)))
--- 214,227 ----
        (dolist (dir (append rmail-movemail-search-path exec-path
  			   (list exec-directory)))
  	(when (and dir (file-accessible-directory-p dir))
! 	  ;; Previously this used to work on Windows by luck (?),
! 	  ;; because rmail-insert-inbox-text fell back to
! 	  ;; (expand-file-name "movemail" exec-directory).
! 	  ;; http://lists.gnu.org/archive/html/bug-gnu-emacs/2008-02/msg00087.html
! 	  (let ((progname (expand-file-name
! 			   (concat "movemail"
! 				   (if (memq system-type '(ms-dos windows-nt))
! 				       ".exe")) dir)))
  	    (when (and (not (file-directory-p progname))
  		       (file-executable-p progname))
  	      (let ((x (rmail-probe progname)))

Re: rmail pop error

[Tim Hesterberg]

The one-line fix works.
The patch works!
Thanks!

Tim

Glenn Morris wrote:
>Tim Hesterberg wrote:
>
>> (rmail-autodetect)
>> returns nil in both 22.1.90 and 22.1.1  (after M-x rmail)
>[...]
>> This does exist, and is executable
>> d:/emacs/emacs-22.1.90/bin/movemail.exe
>
>Thanks; I think I see what's going on now. Does it work in 22.1.90 if
>you add:
>
>(setq rmail-movemail-program "d:/emacs/emacs-22.1.90/bin/movemail.exe")
>
>to your setup?
>
>Without doing that, does this patch fix it? (Apply to rmail.el in
>22.1.90, recompile the file, restart Emacs.)
>
>
>*** rmail.el	31 Jan 2008 10:39:19 -0000	1.433.2.9
>--- rmail.el	20 Feb 2008 02:11:19 -0000
>***************
>*** 214,220 ****
>        (dolist (dir (append rmail-movemail-search-path exec-path
>  			   (list exec-directory)))
>  	(when (and dir (file-accessible-directory-p dir))
>! 	  (let ((progname (expand-file-name "movemail" dir)))
>  	    (when (and (not (file-directory-p progname))
>  		       (file-executable-p progname))
>  	      (let ((x (rmail-probe progname)))
>--- 214,227 ----
>        (dolist (dir (append rmail-movemail-search-path exec-path
>  			   (list exec-directory)))
>  	(when (and dir (file-accessible-directory-p dir))
>! 	  ;; Previously this used to work on Windows by luck (?),
>! 	  ;; because rmail-insert-inbox-text fell back to
>! 	  ;; (expand-file-name "movemail" exec-directory).
>! 	  ;; http://lists.gnu.org/archive/html/bug-gnu-emacs/2008-02/msg00087.html
>! 	  (let ((progname (expand-file-name
>! 			   (concat "movemail"
>! 				   (if (memq system-type '(ms-dos windows-nt))
>! 				       ".exe")) dir)))
>  	    (when (and (not (file-directory-p progname))
>  		       (file-executable-p progname))
>  	      (let ((x (rmail-probe progname)))

Re: rmail pop error

[Glenn Morris]

Tim Hesterberg wrote:

> The one-line fix works.
> The patch works!

And to go back to where we started, does undigestify work ok?

Re: rmail pop error

[Tim Hesterberg]

>Tim Hesterberg wrote:
>
>> The one-line fix works.
>> The patch works!
>
>And to go back to where we started, does undigestify work ok?

Yes, using emacs 22.1.90.1 .

Thanks,
Tim