Problem with imap.el under Windows

Problem with imap.el under Windows

[Nathaniel Calloway]

Hi all.

I'm hoping someone with more experience setting up emacs under windows
will be able to help me figure out what the problem is here. It reaks
of some OS problem, but I don't know enough about what that problem
could be.

I've been trying to fix some problems with gnus for a while now and
corresponding on gnu.emacs.gnus. However, I've narrowed down the
problem to imap.el, so I believe it should be appropriate to post
here. Specifically it is a problem with a set of functions in imap.el,
the imap-ssl-open, imap-tls-open, and similar functions
(imap-BLANK-open, etc, at least each that i've tested). I am putting
this out there in case anyone knows of a patch or other solution. I am
using gnus 5.11 with emacs 22.2, fyi.

To recap the problem, I've been trying to get imap working with gnus
under windows. I'm not a windows person so I don't know what specifics
of windows might be causing a problem in the code. When I try to
connect to a server with any protocol that required an external
process (gssapi, tls, ssl, kerberos), after the process is initialized
(with the imap-ssl-open type function) emacs locks up. This is not a
problem with the external program (the connection is made just fine,
and the process is running, waiting for imap commands) or any certs or
anything else. Yes I am sure of this.

Using imap-tls-open as an example:

Running imap-tls-open locks up emacs, but the function (used inu
imap-tls-open) open-tls-stream works just fine, and starts a working
tls connection from within emacs. The same is true of the other
protocols.

THEREFORE, it is my guess that there is something in common to these
imap.el functions that is breaking emacs.

-Nat

PS - Don't bother asking for logs from gnutls, opensll, imtest,
starttls, etc. They all look fine. If you know how to extract a dump
from emacs stuck in an infinate loop of some type, let me
know. Otherwise I got nothing.

Re: Problem with imap.el under Windows

[Jason Rumney]

On Apr 15, 10:24 pm, Nathaniel Calloway <n...@cornell.edu> wrote:

> Using imap-tls-open as an example:
>
> Running imap-tls-open locks up emacs, but the function (used inu
> imap-tls-open) open-tls-stream works just fine, and starts a working
> tls connection from within emacs. The same is true of the other
> protocols.
>
> THEREFORE, it is my guess that there is something in common to these
> imap.el functions that is breaking emacs.

In the case of gnutls-cli, there seems to be some buffering of output
going on that prevents it being useful as a subprocess, at least in
the latest official Windows build. This is a common problem in Windows
command-line programs, as the default behaviour of the system
libraries changes depending on whether you are running in an
interactive terminal or pipe, so if you don't explicitly flush stdout,
you can never be sure that the output has gone into the pipe.  To
demonstrate, from the command line, "gnutls-cli -p 993 IMAP-SERVER"
outputs:
    ....
    - Handshake was completed

    - Simple Client Mode:

    * OK Dovecot ready.

While C-x b *test* C-: (start-process "test" "*test*" "gnutls-cli" "-
p" "993" "IMAP-SERVER") is missing the last line, and if you
subsequently do (process-send-string "test" "1 CAPABILITY"), the "* OK
Dovecot ready" line appears, but the output of the capability command
does not appear until you issue another command.  Even (process-send-
string "test" "2 LOGOUT") does not appear to terminate the process
until you give it another line of output.

Besides that, Gnus forces the process encoding to binary, so it picks
up all the ^M characters from the server (most internet protocols use
CRLF line ends), but the buffer into which it puts the output does not
have an encoding specified, so it will be created with the default
line-end convention for the platform. Effectively this means that on
Windows, depending on how the information is subsequently read from
that buffer by Gnus, you may end up with a double ^M^M at the end of
each line, which might cause problems somewhere.  I can't get far
enough through to see if this is really a problem, but it might
explain why all the methods that involve external programs have
problems, not just ones that use the broken gnutls-cli.

Re: Problem with imap.el under Windows

[Nathaniel Calloway]

Jason Rumney <jasonrumney@gmail.com> writes:

Thanks so much for your help! You've been the most helpful of anyone
I've asked yet.

> While C-x b *test* C-: (start-process "test" "*test*" "gnutls-cli" "-
> p" "993" "IMAP-SERVER") is missing the last line, and if you
> subsequently do (process-send-string "test" "1 CAPABILITY"), the "* OK
> Dovecot ready" line appears, but the output of the capability command
> does not appear until you issue another command.  Even (process-send-
> string "test" "2 LOGOUT") does not appear to terminate the process
> until you give it another line of output.

Well, crud. Windows was the culprit, it appears. I have replicated the
above, and figured out that this is indeed the problem with the
functions I mentioned.

> Besides that, Gnus forces the process encoding to binary, so it picks
> up all the ^M characters from the server (most internet protocols use
> CRLF line ends), but the buffer into which it puts the output does not
> have an encoding specified, so it will be created with the default
> line-end convention for the platform. Effectively this means that on
> Windows, depending on how the information is subsequently read from
> that buffer by Gnus, you may end up with a double ^M^M at the end of
> each line, which might cause problems somewhere.  I can't get far
> enough through to see if this is really a problem, but it might
> explain why all the methods that involve external programs have
> problems, not just ones that use the broken gnutls-cli.

All the other external programs have the same problem. They are all
waiting for the "OK IMAP server ready" message, so they check once a
second for enternity. Faking a connection and sending the messages
with ^M's back into imap.el didn't cause any problems...yet.


Well, this seems hackable, but do you have any insight into
ways arround this. I will give it a try nonetheless.

Thanks again,

-Nat

Re: Problem with imap.el under Windows

[Jason Rumney]

> Well, this seems hackable, but do you have any insight into
> ways arround this. I will give it a try nonetheless.

You'll need to recompile gnutls-cli (or openssl or whatever) after
inserting the following statement somewhere in its startup sequence:

setvbuf (stdout, NULL, _IONBF, 0);