"no process" gnus bug when moving messages from nnml to imap folder

"no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

There's something broken in Gnus with `B m' in nnml->imap in the latest git
snapshot of Emacs I downloaded and rebuilt last night:

  (version)
  => "GNU Emacs 24.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.20.1) of 2011-02-01 on gkeramidas"

I tried this afternoon to copy some messages from my local nnml archive
to imap folders at Gmail.  Every time I type 'g' the imap server of
Gmail is correctly contacted, and new messages appear fine.  But when I
am trying from the summary buffer of an nnml folder to move a message by
typing `B m' and select an imap folder as the target group, I get the
same backtrace:

    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;; Backtrace from toggle-debug-on-error...

    Debugger entered--Lisp error: (error "Buffer *nnimap imap.gmail.com 993  *nntpd**<4> has no process")
      process-send-string(nil "177 EXAMINE \"archive/2001\"^M\n")
      nnimap-send-command("EXAMINE %S" "archive/2001")
      nnimap-find-article-by-message-id("archive/2001" "<200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>")
      nnimap-request-accept-article("archive/2001" "gmail" t)
      gnus-request-accept-article("nnimap+gmail:archive/2001" nil t t)
      eval((gnus-request-accept-article "nnimap+gmail:archive/2001" (quote nil) t t))
      nnml-request-move-article(2104 "mail.archive-2001" "" (gnus-request-accept-article "nnimap+gmail:archive/2001" (quote nil) t t) t nil)
      gnus-request-move-article(2104 "nnml:mail.archive-2001" "" (gnus-request-accept-article "nnimap+gmail:archive/2001" (quote nil) t t) t nil)
      gnus-summary-move-article(nil)
      call-interactively(gnus-summary-move-article nil nil)

Trying a few things to see what's going on, I could see the following...

    ;; Part 1. Before the attempt that returned the <4> buffer...

    ELISP> (defun gker/buffer-process-alist ()
    	 (do ((buffers (buffer-list) (cdr buffers))
    	      (result nil))
    	     ((null buffers) result)
    	   (let* ((b (car buffers))
    		  (p (or (get-buffer-process b) :no-process)))
    	     (add-to-list 'result (cons b p)))))
    gker/buffer-process-alist
    ELISP> (gker/buffer-process-alist)
    ((#<buffer *nnimap imap.gmail.com 993  *nntpd**<2>> . :no-process)
     (#<buffer  *DOC*> . :no-process)
     (#<buffer  *nnml move*> . :no-process)
     (#<buffer  *copy article*> . :no-process)
     (#<buffer .newsrc-dribble> . :no-process)
     (#<buffer *Article nnml:mail.archive-2001*> . :no-process)
     (#<buffer  *Original Article nnml:mail.archive-2001*> . :no-process)
     (#<buffer *imap log*> . :no-process)
     (#<buffer  *gnus work*> . :no-process)
     (#<buffer *gnus trace*> . :no-process)
     (#<buffer  *Gnus agent overview*> . :no-process)
     (#<buffer  *nntpd*> . :no-process)
     (#<buffer  *Echo Area 1*> . :no-process)
     (#<buffer  *Echo Area 0*> . :no-process)
     (#<buffer  *code-conversion-work*> . :no-process)
     (#<buffer  *Minibuf-0*> . :no-process)
     (#<buffer nnimap.el.gz> . :no-process)
     (#<buffer *nnimap imap.gmail.com 993  *nntpd**> . :no-process)
     (#<buffer *nnimap imap.gmail.com nil  *nntpd**> . :no-process)
     (#<buffer *nnimap imap.gmail.com nil  *nntpd**<2>> . #<process *nnimap*<1>>)
     (#<buffer *Group*> . :no-process)
     (#<buffer *nnimap imap.gmail.com 993  *nntpd**<3>> . :no-process)
     (#<buffer *Ibuffer*> . :no-process)
     (#<buffer *Summary nnml:mail.archive-2001*> . :no-process)
     (#<buffer *Help*> . :no-process)
     (#<buffer *Messages*> . :no-process)
     (#<buffer  *Minibuf-1*> . :no-process)
     (#<buffer *scratch*> . :no-process)
     (#<buffer *ielm*> . #<process ielm>))

    ELISP> (nnimap-buffer)
    #<buffer *nnimap imap.gmail.com 993  *nntpd**<3>>
    ELISP> nnimap-connection-alist
    ((#<buffer  *nntpd*> #<buffer *nnimap imap.gmail.com 993  *nntpd**<3>>))

So at this point `nnimap-connection-alist' is already broken, because it
points to the "*nnimap imap.gmail.com 993  *nntpd**<3>>" buffer, which
has no associated process.  Typing `B m' to move a message throws the
backtrace I showed earlier, and `nnimap-connection-alist' is updated to
point to _another_ buffer, with an increased <4> number.  This new
buffer is also detached from the imap process, so it isn't possible to
do anything with the new `nnimap-connection-alist'.

    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;; Part 2. After trying `B m' to move a message from a local nnml group
    ;; to an imap group at my gmail account, I got the backtrace at the
    ;; start of this message, and...

    ELISP> (nnimap-buffer)
    #<buffer *nnimap imap.gmail.com 993  *nntpd**<4>>

    ELISP> nnimap-connection-alist
    ((#<buffer  *nntpd*> #<buffer *nnimap imap.gmail.com 993  *nntpd**<4>>))

    ;; Now `nnimap-connection-alist' points to the <4> imap buffer, which
    ;; has no associated process/connection.

    ELISP> (gker/buffer-process-alist)
    ((#<buffer *nnimap imap.gmail.com 993  *nntpd**<4>> . :no-process)
     (#<buffer *nnimap imap.gmail.com 993  *nntpd**<2>> . :no-process)
     (#<buffer  *DOC*> . :no-process)
     (#<buffer  *nnml move*> . :no-process)
     (#<buffer  *copy article*> . :no-process)
     (#<buffer .newsrc-dribble> . :no-process)
     (#<buffer *Article nnml:mail.archive-2001*> . :no-process)
     (#<buffer  *Original Article nnml:mail.archive-2001*> . :no-process)
     (#<buffer *imap log*> . :no-process)
     (#<buffer  *gnus work*> . :no-process)
     (#<buffer *gnus trace*> . :no-process)
     (#<buffer  *Gnus agent overview*> . :no-process)
     (#<buffer  *nntpd*> . :no-process)
     (#<buffer  *Echo Area 1*> . :no-process)
     (#<buffer  *Echo Area 0*> . :no-process)
     (#<buffer  *code-conversion-work*> . :no-process)
     (#<buffer  *Minibuf-0*> . :no-process)
     (#<buffer nnimap.el.gz> . :no-process)
     (#<buffer *nnimap imap.gmail.com 993  *nntpd**> . :no-process)
     (#<buffer *nnimap imap.gmail.com nil  *nntpd**> . :no-process)
     (#<buffer *nnimap imap.gmail.com nil  *nntpd**<2>> . #<process *nnimap*<1>>)
     (#<buffer *nnimap imap.gmail.com 993  *nntpd**<3>> . :no-process)
     (#<buffer *Ibuffer*> . :no-process)
     (#<buffer *Help*> . :no-process)
     (#<buffer *Messages*> . :no-process)
     (#<buffer *scratch*> . :no-process)
     (#<buffer *Group*> . :no-process)
     (#<buffer  *Minibuf-1*> . :no-process)
     (#<buffer *Summary nnml:mail.archive-2001*> . :no-process)
     (#<buffer *ielm*> . #<process ielm>))

    ELISP>

For some reason typing `B m' from an nnml group creates new *nnimap ... <N>*
buffers that are *not* associated with the open nnimap connection.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

>       nnimap-send-command("EXAMINE %S" "archive/2001")
>       nnimap-find-article-by-message-id("archive/2001" "<200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M, <200110022311.f92NBXI04280@mailsrv.otenet.gr>")
>       nnimap-request-accept-article("archive/2001" "gmail" t)

That's weird.  Does the message have several Message-ID headers?  Could
you forward the message in question to me, and I'll try to reproduce the
bug.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Thu, 03 Feb 2011 06:18:30 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Giorgos Keramidas <gkeramidas@gmail.com> writes:
>
>>       nnimap-send-command("EXAMINE %S" "archive/2001")
>>       nnimap-find-article-by-message-id("archive/2001"
>> "<200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>^M,
>> <200110022311.f92NBXI04280@mailsrv.otenet.gr>")
>>       nnimap-request-accept-article("archive/2001" "gmail" t)
>
> That's weird.  Does the message have several Message-ID headers?  Could
> you forward the message in question to me, and I'll try to reproduce the
> bug.

It's more weird that just the message-id.  The article has a partially
corrupt body; it's what I managed to salvage off an old email archive.
The text of the article itself has only _one_ message-id header though.

I've attached it in gzip compressed format.

The same problem (no process for buffer *nnimap ... <4>*) happens for
other messages too, however.  It doesn't seem related to the message-id
of the article, but to the way we create nnimap connection buffers.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

> It's more weird that just the message-id.  The article has a partially
> corrupt body; it's what I managed to salvage off an old email archive.
> The text of the article itself has only _one_ message-id header though.
>
> I've attached it in gzip compressed format.

It arrived without an attachment.

> The same problem (no process for buffer *nnimap ... <4>*) happens for
> other messages too, however.  It doesn't seem related to the message-id
> of the article, but to the way we create nnimap connection buffers.

Do you have a backtrace for one of the normal articles?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Thu, 03 Feb 2011 12:27:23 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Giorgos Keramidas <gkeramidas@gmail.com> writes:
>
>> It's more weird that just the message-id.  The article has a partially
>> corrupt body; it's what I managed to salvage off an old email archive.
>> The text of the article itself has only _one_ message-id header though.
>>
>> I've attached it in gzip compressed format.
>
> It arrived without an attachment.
>
>> The same problem (no process for buffer *nnimap ... <4>*) happens for
>> other messages too, however.  It doesn't seem related to the message-id
>> of the article, but to the way we create nnimap connection buffers.
>
> Do you have a backtrace for one of the normal articles?

I'm not sure I understand the question, but I'll try to reply.

I don't get a backtrace when I move articles from a local nnml group to
another, also local, nnml group.  I only get a backtrace when I try to
move messages from a local nnml group to an nnimap destination group.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

>> Do you have a backtrace for one of the normal articles?
>
> I'm not sure I understand the question, but I'll try to reply.
>
> I don't get a backtrace when I move articles from a local nnml group to
> another, also local, nnml group.  I only get a backtrace when I try to
> move messages from a local nnml group to an nnimap destination group.

That message looked a bit special, what with its partially corrupt body
and stuff.

I just don't understand how

(message-field-value "message-id")

returned the string that it seemed to return -- the same Message-ID
several times, with a carriage return in between.

Gmail is hanging up after we've sent the message to Gmail (with APPEND),
and that's most likely because of an encoding issue with the general
brokenness of the message.  Possibly.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

I've verified that I can reproduce the error, but I'm not quite sure
what the solution is.  The message is really corrupt (ending in a lot of
NUL characters), and Gmail just seems to close its connection.  Does
their IMAP server segfault?  I have no idea.

Moving the same message to a Dovecot server works perfectly, so it seems
likely that it's just another shoddy Google product.

Obviously the error message in this case would be better than a
backtrace, and I'll fix that at least.

If anyone has any input on whether Gnus should try to, er, encode the
message in some way before pushing it to Gmail, I'm all ears.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Thu, 03 Feb 2011 13:00:45 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> I've verified that I can reproduce the error, but I'm not quite sure
> what the solution is.  The message is really corrupt (ending in a lot of
> NUL characters), and Gmail just seems to close its connection.  Does
> their IMAP server segfault?  I have no idea.
>
> Moving the same message to a Dovecot server works perfectly, so it seems
> likely that it's just another shoddy Google product.
>
> Obviously the error message in this case would be better than a
> backtrace, and I'll fix that at least.
>
> If anyone has any input on whether Gnus should try to, er, encode the
> message in some way before pushing it to Gmail, I'm all ears.

I think there are two slightly related problems at work here.  One of
them is that Gmail disconnects after we send NUL characters so we can't
move the message and we throw a backtrace.

Another is that once this happens *once* for some message, then it's no
longer possible to `B m' even other normal messages.  It seems that when
Gmail drops the connection Gnus gets 'stuck' trying to recover from the
dropped connection, or trying to use the connection that is no longer
available.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Thu, 03 Feb 2011 13:33:57 -0800, Giorgos Keramidas <gkeramidas@gmail.com> wrote:
> On Thu, 03 Feb 2011 13:00:45 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>> I've verified that I can reproduce the error, but I'm not quite sure
>> what the solution is.  The message is really corrupt (ending in a lot of
>> NUL characters), and Gmail just seems to close its connection.  Does
>> their IMAP server segfault?  I have no idea.
>>
>> Moving the same message to a Dovecot server works perfectly, so it seems
>> likely that it's just another shoddy Google product.
>>
>> Obviously the error message in this case would be better than a
>> backtrace, and I'll fix that at least.
>>
>> If anyone has any input on whether Gnus should try to, er, encode the
>> message in some way before pushing it to Gmail, I'm all ears.
>
> I think there are two slightly related problems at work here.  One of
> them is that Gmail disconnects after we send NUL characters so we can't
> move the message and we throw a backtrace.
>
> Another is that once this happens *once* for some message, then it's no
> longer possible to `B m' even other normal messages.  It seems that when
> Gmail drops the connection Gnus gets 'stuck' trying to recover from the
> dropped connection, or trying to use the connection that is no longer
> available.

Here's what my *Messages* buffer says after one such connection drop,
for example:

  Opening nnimap server on gmail...
  Opening connection to imap.gmail.com via tls...
  Opening TLS connection to `imap.gmail.com'...
  Opening TLS connection with `gnutls-cli --insecure -p 993 imap.gmail.com'...failed
  Opening TLS connection with `gnutls-cli --insecure -p 993 imap.gmail.com --protocols ssl3'...failed
  Opening TLS connection with `openssl s_client -connect imap.gmail.com:993 -no_ssl2 -ign_eof'...done
  Opening TLS connection to `imap.gmail.com'...done
  Opening nnimap server on gmail...done
  Moving to nnimap+gmail:archive/2001: (3135)...
  nnimap-send-command: Buffer *nnimap imap.gmail.com 993  *nntpd**<2> has no process

It looks like Gnus *does* attempt to recover, reconnects to Gmail using
openssl for ssl encryption, and then, for some reason, fails to find the
connection it just opened.

At the same time, my buffer list *does* include at least _one_ valid
connections to Gmail's imap server:

   ,-----------------------------------------------------------------------
   | MR Name                                          Size Mode             Filename/Process
   | -- ----                                          ---- ----             ----------------
   |[ Default ]
   | *  *scratch*                                      463 Lisp Interaction
   | *  *unsent wide reply to Giorgos Keramidas*      2734 Message          ~/News/drafts/drafts/5
   | *% *Summary nnml:posted*                          977 Summary
   | *% *Article nnml:posted*                         1522 Article
   | *% *Group*                                       2331 Group
   | *  *Messages*                                    7852 Fundamental
   | *% *Summary nnml:mail.archive-2001*             78917 Summary
   | *  *gnus trace*                                  2099 Fundamental
   | *  *nnimap imap.gmail.com 993  *nntpd**             0 Fundamental
   | *  *imap log*                                   17217 Fundamental
=> | *  *nnimap imap.gmail.com nil  *nntpd**           491 Fundamental      (*nnimap*<1> run)
   |    .bbdb                                        18944 Fundamental      ~/.bbdb
   |    *sent wide reply to Lars Ingebrigtsen*        1636 Message          ~/News/drafts/drafts/2
   |    *sent mail to Lars Ingebrigtsen*              1808 Message          ~/News/drafts/drafts/4
   |  % *Article nnml:mail.archive-2001*                 0 Article
   | *  *nnimap imap.gmail.com 993  *nntpd**<2>          0 Fundamental
   | *  .newsrc-dribble                              18332 Fundamental      ~/.newsrc-dribble
   |
   |    17 buffers                                  155323                  5 files, 1 process
   `-----------------------------------------------------------------------

The `*nnimap imap.gmail.com 993 *nntpd**' buffer seems disconnected
though.  So does `*nnimap imap.gmail.com 993 *nntpd**<2>'.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Thu, 03 Feb 2011 13:39:31 -0800, Giorgos Keramidas <gkeramidas@gmail.com> wrote:
> Here's what my *Messages* buffer says after one such connection drop,
> for example:

Another thing that may be useful is that the buffer-process problem
_only_ manifests when I try `B m' to move messages from one folder to
another.  If I type `g' in my *Groups* buffer to check for new messages,
a new connection is made to Gmail's imap server and messages are checked
successfully.  My *Messages* buffer then shows:

    Checking new news...
    Opening connection to imap.gmail.com via tls...
    Opening TLS connection to `imap.gmail.com'...
    Opening TLS connection with `gnutls-cli --insecure -p 993 imap.gmail.com'...failed
    Opening TLS connection with `gnutls-cli --insecure -p 993 imap.gmail.com --protocols ssl3'...failed
    Opening TLS connection with `openssl s_client -connect imap.gmail.com:993 -no_ssl2 -ign_eof'...done
    Opening TLS connection to `imap.gmail.com'...done
    Reading active file via nnml...
    Opening nnml server...done
    Reading incoming mail from file...
    Wrote /home/gkeramidas/Mail/mail/freebsd/questions/12886
    nnml: Reading incoming mail (1 new)...done
    Reading active file via nnml...done
    nnimap read 0k [7 times]
    nnimap read 1k [5 times]
    nnimap read 2k [6 times]
    nnimap read 3k [2 times]
    nnimap read 4k [2 times]
    nnimap read 5k [7 times]
    nnimap read 6k [6 times]
    nnimap read 7k
    nnimap read 8k [2 times]
    nnimap read 9k [2 times]
    nnimap read 10k [8 times]
    nnimap read 11k [17 times]
    nnimap read 12k [3 times]
    nnimap read 13k [4 times]
    nnimap read 15k [8 times]
    nnimap read 16k [4 times]
    nnimap read 17k
    nnimap read 18k [2 times]
    nnimap read 19k [9 times]
    nnimap read 21k
    nnimap read 22k
    nnimap read 23k [9 times]
    nnimap read 24k [12 times]
    nnimap read 25k [7 times]
    nnimap read 26k [2 times]
    nnimap read 27k [2 times]
    nnimap read 28k [3 times]
    nnimap read 29k [10 times]
    nnimap read 30k [9 times]
    Reading active file via nndraft...done
    Checking new news...done
    Retrieving newsgroup: nnml:mail.INBOX...
    Fetching headers for nnml:mail.INBOX...done
    Scoring...done
    Scoring...done
    Generating summary...done
    No more unseen articles
    Expiring articles...done

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Eli Zaretskii]

> From: Giorgos Keramidas <gkeramidas@gmail.com>
> Date: Thu, 03 Feb 2011 13:33:57 -0800
> Cc: ding@gnus.org, emacs-devel@gnu.org
> 
> On Thu, 03 Feb 2011 13:00:45 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> > I've verified that I can reproduce the error, but I'm not quite sure
> > what the solution is.  The message is really corrupt (ending in a lot of
> > NUL characters), and Gmail just seems to close its connection.  Does
> > their IMAP server segfault?  I have no idea.
> >
> > Moving the same message to a Dovecot server works perfectly, so it seems
> > likely that it's just another shoddy Google product.
> >
> > Obviously the error message in this case would be better than a
> > backtrace, and I'll fix that at least.
> >
> > If anyone has any input on whether Gnus should try to, er, encode the
> > message in some way before pushing it to Gmail, I'm all ears.
> 
> I think there are two slightly related problems at work here.  One of
> them is that Gmail disconnects after we send NUL characters so we can't
> move the message and we throw a backtrace.

Don't some network-related functions return null characters if you try
to read from a socket after the connection is closed?

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Fri, 04 Feb 2011 10:38:57 +0200, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Giorgos Keramidas <gkeramidas@gmail.com>
>> Date: Thu, 03 Feb 2011 13:33:57 -0800
>> Cc: ding@gnus.org, emacs-devel@gnu.org
>>
>> On Thu, 03 Feb 2011 13:00:45 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>> > I've verified that I can reproduce the error, but I'm not quite sure
>> > what the solution is.  The message is really corrupt (ending in a lot of
>> > NUL characters), and Gmail just seems to close its connection.  Does
>> > their IMAP server segfault?  I have no idea.
>> >
>> > Moving the same message to a Dovecot server works perfectly, so it seems
>> > likely that it's just another shoddy Google product.
>> >
>> > Obviously the error message in this case would be better than a
>> > backtrace, and I'll fix that at least.
>> >
>> > If anyone has any input on whether Gnus should try to, er, encode the
>> > message in some way before pushing it to Gmail, I'm all ears.
>>
>> I think there are two slightly related problems at work here.  One of
>> them is that Gmail disconnects after we send NUL characters so we can't
>> move the message and we throw a backtrace.
>
> Don't some network-related functions return null characters if you try
> to read from a socket after the connection is closed?

I think they return zero as their status-value but not extra ascii bytes
whose value is zero.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Don't some network-related functions return null characters if you try
> to read from a socket after the connection is closed?

Sounds likely.

I've verified that putting a single NUL character into any mail and then
trying to move it to Gmail will make Gmail hang up the connection.

So the Gmail IMAP servers just aren't compliant -- the IMAP RFC says
that all IMAP servers should be able to take binary data after APPEND, I
seem to recall.

The question is -- what should nnimap do about that?  Introduce a
"quirks mode" for Gmail that just strips all NUL characters?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

> Another is that once this happens *once* for some message, then it's no
> longer possible to `B m' even other normal messages.  It seems that when
> Gmail drops the connection Gnus gets 'stuck' trying to recover from the
> dropped connection, or trying to use the connection that is no longer
> available.

Do you get the same result if you use `B c' instead of `B m'?  I've been
testing with `B c' (since copying is easier -- you can copy as many
times as you want), and I am unable to reproduce this.  If I `B c' a
NUL-bearing message, Gmail drops the connection, and if I then `B c' a
non-NUL message, everything works just fine -- it reestablishes the
connection and copies the message.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Ted Zlatanov]

On Fri, 04 Feb 2011 00:53:31 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> I've verified that putting a single NUL character into any mail and then
LI> trying to move it to Gmail will make Gmail hang up the connection.

LI> So the Gmail IMAP servers just aren't compliant -- the IMAP RFC says
LI> that all IMAP servers should be able to take binary data after APPEND, I
LI> seem to recall.

LI> The question is -- what should nnimap do about that?  Introduce a
LI> "quirks mode" for Gmail that just strips all NUL characters?

I would ask the use how to treat the NULs, similarly to how Gnus asks
about unprintable characters in an outgoing e-mail.

Ted

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I would ask the use how to treat the NULs, similarly to how Gnus asks
> about unprintable characters in an outgoing e-mail.

But only for Gmail servers, I guess?  None of the other I've tried seem
to have any problems with NULs.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Fri, 04 Feb 2011 00:58:35 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Giorgos Keramidas <gkeramidas@gmail.com> writes:
>
>> Another is that once this happens *once* for some message, then it's no
>> longer possible to `B m' even other normal messages.  It seems that when
>> Gmail drops the connection Gnus gets 'stuck' trying to recover from the
>> dropped connection, or trying to use the connection that is no longer
>> available.
>
> Do you get the same result if you use `B c' instead of `B m'?  I've
> been testing with `B c' (since copying is easier -- you can copy as
> many times as you want), and I am unable to reproduce this.  If I `B
> c' a NUL-bearing message, Gmail drops the connection, and if I then `B
> c' a non-NUL message, everything works just fine -- it reestablishes
> the connection and copies the message.

Yes, I get the same result with `B c'.  What seems odd is that I see two
different imap backends in my buffer list.  My ~/.gnus file includes:

  (setq gnus-nntp-server "news.sunsite.dk")
  (setq nntp-authinfo-function 'nntp-send-authinfo)
  (setq gnus-select-method '(nntp "news.sunsite.dk"))
  (setq gnus-secondary-select-methods '((nnml "")
                                        (nnimap "gmail"
                                                (nnimap-address "imap.gmail.com")
                                                (nnimap-server-port 993)
                                                (nnimap-stream tls))))

Completion offers both `nnimap:gmail' and `nnimap:imap.gmail.com' as a
valid `B c' destination, but they both respond that the newly created
imap buffer has no connection when I move or copy messages from an nnml
source folder.

Other imap operations seem to work fine though, e.g. all the following
work fine:

  - copying messages from imap folders to local nnml folders
  - moving messages from imap folders to local nnml folders
  - getting new messages into imap folders by typing `g'
  - flag updates when I modify messages through gmail's web interface
    and type `g' in gnus

I can even see messages marked in Gnus with '!' as starred messages in
the web interface of Gmail, and when I unmark them in Gnus and type `g'
they get unstarred in the web UI too.

Whatever the problem is with `B c' and `B m' it seems specific to using
an imap folder as the destination of the messages.  As a source nnimap
folders seem to work more than fine.  I noticed that nnimap access is
now blazingly fast too.  I started using Gmail's imap access method only
because it's now very very fast to access the messages from Gnus too :-)

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Ted Zlatanov]

On Fri, 04 Feb 2011 10:08:40 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I would ask the use how to treat the NULs, similarly to how Gnus asks
>> about unprintable characters in an outgoing e-mail.

LI> But only for Gmail servers, I guess?  None of the other I've tried seem
LI> to have any problems with NULs.

Sure.  It may not work to autodetect gmail.com with Google Apps, if
those support the IMAP access, since they run under another domain name.
Can someone knowledgeable about that comment?  If my suspicion is right,
it would have to be a backend slot variable.

Ted

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Sure.  It may not work to autodetect gmail.com with Google Apps, if
> those support the IMAP access, since they run under another domain name.
> Can someone knowledgeable about that comment?  If my suspicion is right,
> it would have to be a backend slot variable.

The greeting is

* OK Gimap ready for requests from 198.144.208.130 k9if2067598fax.47

so if we just match for "OK Gimap", then we should probably be safe.

I'm wondering whether to over-engineer this by adding a quirks mode
mechanism, or whether to just special-case this in the -request-accept
function, or, hm.

I'd really prefer that Gnus queried the user instead of the backend, but
Gnus knows absolutely nothing about this, so I guess it would make most
sense to just add a couple of if statements to nnimap-request-accept,
even though it's kinda uglyish.  From my viewpoint as a licenced Over
Engineer.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Ted Zlatanov]

On Fri, 04 Feb 2011 11:14:18 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Sure.  It may not work to autodetect gmail.com with Google Apps, if
>> those support the IMAP access, since they run under another domain name.
>> Can someone knowledgeable about that comment?  If my suspicion is right,
>> it would have to be a backend slot variable.

LI> The greeting is

LI> * OK Gimap ready for requests from 198.144.208.130 k9if2067598fax.47

LI> so if we just match for "OK Gimap", then we should probably be safe.

Sounds good.

LI> I'm wondering whether to over-engineer this by adding a quirks mode
LI> mechanism, or whether to just special-case this in the -request-accept
LI> function, or, hm.

LI> I'd really prefer that Gnus queried the user instead of the backend, but
LI> Gnus knows absolutely nothing about this, so I guess it would make most
LI> sense to just add a couple of if statements to nnimap-request-accept,
LI> even though it's kinda uglyish.  From my viewpoint as a licenced Over
LI> Engineer.

If it's a single special case, do whatever is simplest.  We can
over-engineer it when there's another such case.

Ted

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LI> so if we just match for "OK Gimap", then we should probably be safe.
>
> Sounds good.

I've now implemented this, and it seems to work OK.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

> Completion offers both `nnimap:gmail' and `nnimap:imap.gmail.com' as a
> valid `B c' destination, but they both respond that the newly created
> imap buffer has no connection when I move or copy messages from an nnml
> source folder.

These are just two names for the same server, probably.

> Whatever the problem is with `B c' and `B m' it seems specific to using
> an imap folder as the destination of the messages.

No, as I said previously, it's a bug in Gmail.  It fails when accepting
articles that contain NULs.  I've now added a quirk for this in nnimap.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Mon, 07 Feb 2011 03:42:16 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>Giorgos Keramidas <gkeramidas@gmail.com> writes:
>> Completion offers both `nnimap:gmail' and `nnimap:imap.gmail.com' as a
>> valid `B c' destination, but they both respond that the newly created
>> imap buffer has no connection when I move or copy messages from an nnml
>> source folder.
>
> These are just two names for the same server, probably.
>
>> Whatever the problem is with `B c' and `B m' it seems specific to using
>> an imap folder as the destination of the messages.
>
> No, as I said previously, it's a bug in Gmail.  It fails when accepting
> articles that contain NULs.  I've now added a quirk for this in nnimap.

Ok, thanks.  Is the quirk in the bzr repo?  If yes, my daily fetch from
the git mirror will pick it up and I'll give it a spin again :)

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

> Ok, thanks.  Is the quirk in the bzr repo?

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Giorgos Keramidas]

On Mon, 07 Feb 2011 10:45:01 -0800, Giorgos Keramidas <gkeramidas@gmail.com> wrote:
> On Mon, 07 Feb 2011 03:42:16 -0800, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>>Giorgos Keramidas <gkeramidas@gmail.com> writes:
>>> Completion offers both `nnimap:gmail' and `nnimap:imap.gmail.com' as a
>>> valid `B c' destination, but they both respond that the newly created
>>> imap buffer has no connection when I move or copy messages from an nnml
>>> source folder.
>>
>> These are just two names for the same server, probably.
>>
>>> Whatever the problem is with `B c' and `B m' it seems specific to using
>>> an imap folder as the destination of the messages.
>>
>> No, as I said previously, it's a bug in Gmail.  It fails when accepting
>> articles that contain NULs.  I've now added a quirk for this in nnimap.
>
> Ok, thanks.  Is the quirk in the bzr repo?  If yes, my daily fetch from
> the git mirror will pick it up and I'll give it a spin again :)

Ok, I've rebuilt Emacs and Gnus from a later version [GNU Emacs
24.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.20.1) of 2011-02-11
on gkeramidas-glaptop].

It still fails to copy my old archived messages to Gmail imap, though.

The message that fails this time is (copied straight from my nnml folder
file at `~/mail/mail/archive/2001/3138'):

  http://people.freebsd.org/~keramida/3138.nnml

When I try to 'B c' this message to `nnimap+gmail:archive/2001' Gnus
fails to copy the message, but there are no NUL bytes in the message
itself.

So as an experiment I saved this message typing `O m' to `~/foo.mbox'
and used Mutt to copy the same message to:

  imaps://gkeramidas@imap.gmail.com/archive/2001

and the copy worked fine.  So I'm not sure if there's something wrong
with the message.  There's definitely something wrong with Gnus when I
try to open a local nnml groupand send messages to imap though.

Re: "no process" gnus bug when moving messages from nnml to imap folder

[Lars Ingebrigtsen]

Giorgos Keramidas <gkeramidas@gmail.com> writes:

> The message that fails this time is (copied straight from my nnml folder
> file at `~/mail/mail/archive/2001/3138'):
>
>   http://people.freebsd.org/~keramida/3138.nnml
>
> When I try to 'B c' this message to `nnimap+gmail:archive/2001' Gnus
> fails to copy the message, but there are no NUL bytes in the message
> itself.

I'm unable to reproduce this bug.  I'm able to copy that article to
Gmail just fine with the current Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen