Very basic question regarding encoding and `open-network-stream'

Very basic question regarding encoding and `open-network-stream'

[Eric Abrahamsen]

I have an embarrassingly basic question regarding how encoding works
with relationship to `open-network-stream'. I don't have a strong grasp
of encoding issues, particularly regarding processes.

Long story short, I'm trying to get Gnus to internally use decoded group
names as much as possible. I'm investigating whether that means that the
process Gnus uses to talk to remote servers should be encoded differently.

Presently, when Gnus talks to an nntp server, it does so in a process
buffer in which multibyte has been disabled. It wraps the call to
`open-network-stream' in a let which sets `coding-system-for-read/write'
to 'binary. So far so clear.

Gnus also mostly leaves group names as unibyte internally, so reading
group names as bytes works out okay -- but this is what I'm looking at
changing.

The NNTP RFC notes that the default character set for the protocol has
changed from ascii to utf-8.
(https://tools.ietf.org/html/rfc3977#section-1)

What I don't grasp is: if the process buffers are left multibyte, and
the `coding-system-for-read/write' variables are changed to 'utf-8 (or
'undecided?), will this simply do the right thing?

More specifically, is the remote nntp server expected to send along some
information about the encoding it is using for its data? Or is the data
always binary, and we simply know via convention that it can be safely
decoded as 'utf-8?

Or maybe I should just be leaving the process buffer as-is, but doing
the decoding immediately after the `accept-process-output'?

I would very much appreciate it if someone could explain step-by-step,
using small words if possible, how the process encoding is negotiated,
and what might be a reasonable approach to this problem.

Thanks,
Eric

Re: Very basic question regarding encoding and `open-network-stream'

[Stefan Monnier]

> The NNTP RFC notes that the default character set for the protocol has
> changed from ascii to utf-8.
> (https://tools.ietf.org/html/rfc3977#section-1)
>
> What I don't grasp is: if the process buffers are left multibyte, and
> the `coding-system-for-read/write' variables are changed to 'utf-8 (or
> 'undecided?), will this simply do the right thing?

No: the stream of bytes includes NNTP protocol commands as well as other
contents (typically actual messages) and they don't all use utf-8.
So the stream process needs to communicate in bytes (aka "unibyte"), and
then the Elisp code needs to decode/encode each part manually according
to the coding system that applies to each part.


        Stefan

Re: Very basic question regarding encoding and `open-network-stream'

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The NNTP RFC notes that the default character set for the protocol has
>> changed from ascii to utf-8.
>> (https://tools.ietf.org/html/rfc3977#section-1)
>>
>> What I don't grasp is: if the process buffers are left multibyte, and
>> the `coding-system-for-read/write' variables are changed to 'utf-8 (or
>> 'undecided?), will this simply do the right thing?
>
> No: the stream of bytes includes NNTP protocol commands as well as other
> contents (typically actual messages) and they don't all use utf-8.
> So the stream process needs to communicate in bytes (aka "unibyte"), and
> then the Elisp code needs to decode/encode each part manually according
> to the coding system that applies to each part.

Awesome, this will save me wandering down some garden paths, thank you.
I will make the Minimum Viable Changes, and just focus on group names.