determining process coding

determining process coding

[Eric Abrahamsen]

I've been trying for a while to improve Gnus' ability to search for
multibyte strings on IMAP servers, by sending in a charset and a string
literal as part of the SEARCH command. So instead of this:

. UID SEARCH SUBJECT "metal band"

we can send this:

. UID SEARCH CHARSET UTF-8 SUBJECT {10+}
Motörhead

The details aren't terribly interesting, basically you specify a
charset, and the number of string bytes in brackets, then send the
string itself on a separate line.

I'm having two problems: 1) I don't really know how to determine the
proper charset to use, and 2) some IMAP servers (*cough* exchange
*cough*) don't accept many charsets, including utf-8.

Number 2 can only be worked around, but number 1 should be resolvable.

Gnus opens a long-running connection to an IMAP server through a process
buffer, and while playing with the code, I tried (process-coding-system
(get-buffer-process (current-buffer))) while the IMAP buffer was
current, and it gave me (binary . binary). I'm not sure what to do with
that information. Searching this very server using utf-8 works just
fine, so I'm assuming that I'm coming to some sort of fallback value.

Can anyone tell me how to reliably tell what coding system Gnus will use
to send strings (with `process-send-string') to the IMAP server?

Bonus points for any wisdom regarding dealing with IMAP servers that
don't accept UTF-8.

Thanks for any pointers,
Eric

Re: determining process coding

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Thu, 12 Feb 2015 18:24:16 +0800
> 
> Gnus opens a long-running connection to an IMAP server through a process
> buffer, and while playing with the code, I tried (process-coding-system
> (get-buffer-process (current-buffer))) while the IMAP buffer was
> current, and it gave me (binary . binary).

This probably means Gnus encodes text "by hand" before sending it.
Look for encode-coding somewhere.

> Bonus points for any wisdom regarding dealing with IMAP servers that
> don't accept UTF-8.

Do they accept UTF-16?

Re: determining process coding

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Thu, 12 Feb 2015 18:24:16 +0800
>> 
>> Gnus opens a long-running connection to an IMAP server through a process
>> buffer, and while playing with the code, I tried (process-coding-system
>> (get-buffer-process (current-buffer))) while the IMAP buffer was
>> current, and it gave me (binary . binary).
>
> This probably means Gnus encodes text "by hand" before sending it.
> Look for encode-coding somewhere.

It turns out that, in `nnimap-open-connection-1', Gnus explicitly lets
`coding-system-for-read|write' to 'binary. The actual connection is
created inside that let, with `open-protocol-stream' (aka
`make-network-stream').

I really don't see much else relevant to coding. Most commands are sent
with `nnimap-send-command', which simply runs:

(process-send-string (get-buffer-process (current-buffer)) COMMAND)

The only de|encoding I see going on is with IMAP folder names, with the
IMAP variant of UTF-7.

>> Bonus points for any wisdom regarding dealing with IMAP servers that
>> don't accept UTF-8.
>
> Do they accept UTF-16?

The culprit here is Exchange 2003 servers. I don't have an account to
test with, so I'm trying to do as much background research as possible
before pestering users for help. Apparently they only accept US-ASCII as
a charset (but I don't see how that could be) -- at least, UTF-8 doesn't
work. I signed up for an outlook.com address to play with, but that
server (imap-mail.outlook.com) is much better behaved than the Exchange
one, and seems to accept UTF-8 just fine.

Thanks,
Eric

Re: determining process coding

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Fri, 13 Feb 2015 12:27:04 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> >> Date: Thu, 12 Feb 2015 18:24:16 +0800
> >> 
> >> Gnus opens a long-running connection to an IMAP server through a process
> >> buffer, and while playing with the code, I tried (process-coding-system
> >> (get-buffer-process (current-buffer))) while the IMAP buffer was
> >> current, and it gave me (binary . binary).
> >
> > This probably means Gnus encodes text "by hand" before sending it.
> > Look for encode-coding somewhere.
> 
> It turns out that, in `nnimap-open-connection-1', Gnus explicitly lets
> `coding-system-for-read|write' to 'binary. The actual connection is
> created inside that let, with `open-protocol-stream' (aka
> `make-network-stream').

That's expected.

> I really don't see much else relevant to coding. Most commands are sent
> with `nnimap-send-command', which simply runs:
> 
> (process-send-string (get-buffer-process (current-buffer)) COMMAND)
> 
> The only de|encoding I see going on is with IMAP folder names, with the
> IMAP variant of UTF-7.

Look in the places that create the strings which are then sent with
nnimap-send-command.  If they are encoded, they will be encoded where
they are consed.

Re: determining process coding

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Fri, 13 Feb 2015 12:27:04 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> >> Date: Thu, 12 Feb 2015 18:24:16 +0800
>> >> 
>> >> Gnus opens a long-running connection to an IMAP server through a process
>> >> buffer, and while playing with the code, I tried (process-coding-system
>> >> (get-buffer-process (current-buffer))) while the IMAP buffer was
>> >> current, and it gave me (binary . binary).
>> >
>> > This probably means Gnus encodes text "by hand" before sending it.
>> > Look for encode-coding somewhere.
>> 
>> It turns out that, in `nnimap-open-connection-1', Gnus explicitly lets
>> `coding-system-for-read|write' to 'binary. The actual connection is
>> created inside that let, with `open-protocol-stream' (aka
>> `make-network-stream').
>
> That's expected.
>
>> I really don't see much else relevant to coding. Most commands are sent
>> with `nnimap-send-command', which simply runs:
>> 
>> (process-send-string (get-buffer-process (current-buffer)) COMMAND)
>> 
>> The only de|encoding I see going on is with IMAP folder names, with the
>> IMAP variant of UTF-7.
>
> Look in the places that create the strings which are then sent with
> nnimap-send-command.  If they are encoded, they will be encoded where
> they are consed.

I'm feeling dense here, but really not seeing anything. All the strings
are just fed to `nnimap-send-command' as they are.

If I'm not wrong about that, what's the most reliable way to figure out
what encoding will end up getting used when talking to the server?

Thanks again,
Eric

Re: determining process coding

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Wed, 18 Feb 2015 11:49:02 +0800
> 
> > Look in the places that create the strings which are then sent with
> > nnimap-send-command.  If they are encoded, they will be encoded where
> > they are consed.
> 
> I'm feeling dense here, but really not seeing anything. All the strings
> are just fed to `nnimap-send-command' as they are.

I suggested to look higher in the calling sequence.

If that doesn't help, I suggest to ask on the Gnus mailing list, where
you'll find people who know how this stuff works, and will point you
to the right place.

> If I'm not wrong about that, what's the most reliable way to figure out
> what encoding will end up getting used when talking to the server?

I was trying to help you figure that out.  I'm sorry, I don't know
Gnus and nnimap well enough to do better.

Re: determining process coding

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Wed, 18 Feb 2015 11:49:02 +0800
>> 
>> > Look in the places that create the strings which are then sent with
>> > nnimap-send-command.  If they are encoded, they will be encoded where
>> > they are consed.
>> 
>> I'm feeling dense here, but really not seeing anything. All the strings
>> are just fed to `nnimap-send-command' as they are.
>
> I suggested to look higher in the calling sequence.
>
> If that doesn't help, I suggest to ask on the Gnus mailing list, where
> you'll find people who know how this stuff works, and will point you
> to the right place.
>
>> If I'm not wrong about that, what's the most reliable way to figure out
>> what encoding will end up getting used when talking to the server?
>
> I was trying to help you figure that out.  I'm sorry, I don't know
> Gnus and nnimap well enough to do better.

No worries, it was just a general question, and I assumed I was making
some dumb assumptions somewhere. I'll try on gnus.general.

Thanks,
Eric