Re: How to get buffer byte length (not number of characters)?

[Eli Zaretskii <eliz@gnu.org>]

> From: Joseph Turner <joseph@ushin.org>
> Cc: emacs-devel@gnu.org,  schwab@suse.de,  adam@alphapapa.net
> Date: Wed, 21 Aug 2024 16:52:39 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Currently, plz.el always creates the curl subprocess like so:
> >> 
> >> (make-process :coding 'binary ...)
> >> 
> >> https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/plz.el?h=externals-release/plz#n519
> >> 
> >> Does this DTRT?
> >
> > It could be TRT if plz.el encodes the buffer text "by hand" before
> > sending the results to curl and decodes it when it receives text from
> > curl.  Which I think is what happens there.
> 
> plz.el does not manually encode buffer text *within Emacs* when sending
> requests to curl, but by default, plz.el sends data to curl with --data,
> which tells curl to strip CR and newlines.  With the :body-type 'binary
> argument, plz.el instead uses --data-binary, which does no conversion.

Newlines is a relatively minor issue (although it, too, needs to be
considered).  My main concern is with the text encoding.  How can it
be TRT to use 'binary when sending buffer text to curl? that would
mean we are more-or-less always sending the internal representation of
characters, which is superset of UTF-8.  If the data was originally
encoded in anything but UTF-8, reading it into Emacs and then sending
it back will change the byte sequences from that other encoding to
UTF-8.  Moreover, 'binary does not guarantee that the result is valid
UTF-8.

So maybe I misunderstand how these plz.el facilities are used, but up
front this sounds like a mistake.

> We don't want to strip newlines from hyperdrive files, so we always use
> :body-type 'binary when sending buffer contents.  Should hyperdrive.el
> encode data with `buffer-file-coding-system' before passing to plz.el?

I would think so, but maybe we should bring the plz.el developers on
board of this discussion.

> When receiving text from curl, plz.el optionally decodes the text
> according to the charset in the 'Content-Type' header, e.g., "text/html;
> charset=utf-8" or utf-8 if no charset is found.

By "optionally" you mean that it doesn't always happen, except if the
caller requests that?  If so, the caller of plz.el should decode the
text manually before using it in user-facing features.

> Perhaps hyperdrive.el should check the 'Content-Type' header charset,
> then fallback to guessing the coding system based on filename and file
> contents with `set-auto-coding' (to avoid decoding images, etc.), and
> then finally fallback to something else?

Probably.  But then I don't know anything about hyperdrive.el, either.
If it copies text between files or URLs without showing it to the
user, then the best strategy is indeed not to decode and encode stuff,
but handle it as a stream of raw bytes.  (In that case, my suggestion
would be to use unibyte buffers and strings for temporarily storing
and processing these raw bytes in Emacs.)  But if the text is somehow
shown to the user, it must be decoded to be displayed correctly by
Emacs.  And then it must be encoded back when writing it back to the
external storage.