bug#38587: base64-decode-region breaks encoding

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

0. emacs -Q
1. insert a non-ASCII char, e.g. ä
2. select the region around the char
3. M-x base64-encode-region
4. select the region around the encoded text
5. M-x base64-decode-region

results in a broken text.  IOW, base64-encode-region and base64-decode-region
are not reversible, whereas their string counterparts are:

(base64-decode-string (base64-encode-string "ä"))
=> "\344"

(the latter expression returns the right result, but inserts broken text too)

bug#38587: base64-decode-region breaks encoding

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> 0. emacs -Q
> 1. insert a non-ASCII char, e.g. ä
> 2. select the region around the char
> 3. M-x base64-encode-region
> 4. select the region around the encoded text
> 5. M-x base64-decode-region
>
> results in a broken text.  IOW, base64-encode-region and base64-decode-region
> are not reversible, whereas their string counterparts are:
>
> (base64-decode-string (base64-encode-string "ä"))
> => "\344"

Well, that's not really reversible, either.

> (the latter expression returns the right result, but inserts broken text too)

None of these functions work on multibyte text (by design), but I see
the doc strings don't mention this.  (The manual does.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38587: base64-decode-region breaks encoding

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 13 Dec 2019 03:52:46 +0100
> Cc: 38587@debbugs.gnu.org
> 
> > (base64-decode-string (base64-encode-string "ä"))
> > => "\344"
> 
> Well, that's not really reversible, either.
> 
> > (the latter expression returns the right result, but inserts broken text too)
> 
> None of these functions work on multibyte text (by design)

Right.

> but I see the doc strings don't mention this.  (The manual does.)

Let's say that in the doc strings as well.  It is not easy to come up
with the right text, btw, because saying just "region must be unibyte"
is inaccurate; see the source of the implementation for the details.
That's why the ELisp manual also doesn't say anything simple in this
respect.

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

>> 0. emacs -Q
>> 1. insert a non-ASCII char, e.g. ä
>> 2. select the region around the char
>> 3. M-x base64-encode-region
>> 4. select the region around the encoded text
>> 5. M-x base64-decode-region
>>
>> results in a broken text.  IOW, base64-encode-region and base64-decode-region
>> are not reversible, whereas their string counterparts are:
>>
>> (base64-decode-string (base64-encode-string "ä"))
>> => "\344"
>
> Well, that's not really reversible, either.

But when it's know that the source string was in UTF-8,
shouldn't it be reversible?  What is needed for that?

Maybe an additional CODING arg for base64-decode-region?

Or it would be enough to use the coding system of the
output buffer?

bug#38587: base64-decode-region breaks encoding

[Andreas Schwab]

On Dez 15 2019, Juri Linkov wrote:

> Maybe an additional CODING arg for base64-decode-region?

BASE64 is defined on a sequence of bytes.  It doesn't make sense to
apply it to characters.

The input of base64-encode-region needs to be encoded into bytes and the
output of base64-decode-region needs to be decoded into characters.  If
you do that, you get a full reversible operation.

> Or it would be enough to use the coding system of the
> output buffer?

The coding system of the output buffer has nothing to do with the coding
of the data produced by base64-decode-region, just like
process-coding-system is independent from the coding system of the
process buffer.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

>> Maybe an additional CODING arg for base64-decode-region?
>
> BASE64 is defined on a sequence of bytes.  It doesn't make sense to
> apply it to characters.

But isn't UTF-8 a multibyte encoding represented by a sequence of bytes
(e.g. when saved to a file)?

Then why base64-encode-region couldn't use the buffer's coding
to convert the region to a sequence of bytes?

Also why base64-encode-region accepts region's characters
only from the charsets ‘eight-bit-control’ and ‘eight-bit-graphic’,
but not other UTF-8 characters?

> The input of base64-encode-region needs to be encoded into bytes and the
> output of base64-decode-region needs to be decoded into characters.  If
> you do that, you get a full reversible operation.

I guess base64-encode-region already encodes the region into bytes,
but only partially - it signals an error on some characters,
I don't understand why it can't encode all of them.

>> Or it would be enough to use the coding system of the
>> output buffer?
>
> The coding system of the output buffer has nothing to do with the coding
> of the data produced by base64-decode-region, just like
> process-coding-system is independent from the coding system of the
> process buffer.

It's understandable that the coding system of the output buffer
is not necessarily the same as expected from the output of
base64-decode-region.

But is it still possible to tell base64-decode-region
about the expected output coding system?  Maybe using
a prefix arg: C-u M-x base64-decode-region could ask
for a coding, defaulting to the buffer's coding.

For example, in Ruby

  require 'base64'
  Base64.decode64(Base64.encode64("☃"))
  => "\xE2\x98\x83"

indeed outputs ASCII not encoded to UTF-8.
But it's possible to force encoding with:

  Base64.decode64(Base64.encode64("☃")).force_encoding('UTF-8')
  => "☃"

Is there an equivalent of force_encoding('UTF-8') in Emacs?
I tried to call after base64-decode-region on its output:

  (decode-coding-region (point-min) (point-max) 'binary)

but it doesn't work, neither this:

  (encode-coding-region (point-min) (point-max) 'utf-8)

Also this doesn't work on the string output:

  (decode-coding-string (base64-decode-string (base64-encode-string "ä"))
                        'utf-8)

Maybe I'm doing something wrong?

bug#38587: base64-decode-region breaks encoding

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Mon, 16 Dec 2019 00:40:55 +0200
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 38587@debbugs.gnu.org
> 
> > BASE64 is defined on a sequence of bytes.  It doesn't make sense to
> > apply it to characters.
> 
> But isn't UTF-8 a multibyte encoding represented by a sequence of bytes
> (e.g. when saved to a file)?

When saved to a file, yes.

> Then why base64-encode-region couldn't use the buffer's coding
> to convert the region to a sequence of bytes?

Because it isn't guaranteed that the buffer's encoding is indeed the
right one for this job.

> Also why base64-encode-region accepts region's characters
> only from the charsets ‘eight-bit-control’ and ‘eight-bit-graphic’,
> but not other UTF-8 characters?

Because it wants raw bytes, and only eight-bit charsets fit that
condition.  Eight-bit charset is the charset of raw bytes in a
multibyte buffer or string.

(base64-encode-region can also work on unibyte buffers and strings, in
which case "charset" of such "text" has no meaning.)

> > The input of base64-encode-region needs to be encoded into bytes and the
> > output of base64-decode-region needs to be decoded into characters.  If
> > you do that, you get a full reversible operation.
> 
> I guess base64-encode-region already encodes the region into bytes,
> but only partially - it signals an error on some characters,
> I don't understand why it can't encode all of them.

Once again, because it wants to process only raw bytes.

> But is it still possible to tell base64-decode-region
> about the expected output coding system?  Maybe using
> a prefix arg: C-u M-x base64-decode-region could ask
> for a coding, defaulting to the buffer's coding.

If we want to make such a change, then "C-x RET c" is a better prefix
command, as it is consistent with other commands that accept
coding-system overrides.

> Is there an equivalent of force_encoding('UTF-8') in Emacs?

"C-x RET c utf-8 RET M-x SOME-COMMAND RET"

> Also this doesn't work on the string output:
> 
>   (decode-coding-string (base64-decode-string (base64-encode-string "ä"))
>                         'utf-8)

It will work if you encode "ä" first:

  (decode-coding-string (base64-decode-string
                         (base64-encode-string
			  (encode-coding-string "ä" 'utf-8)))
			'utf-8)

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

>> But is it still possible to tell base64-decode-region
>> about the expected output coding system?  Maybe using
>> a prefix arg: C-u M-x base64-decode-region could ask
>> for a coding, defaulting to the buffer's coding.
>
> If we want to make such a change, then "C-x RET c" is a better prefix
> command, as it is consistent with other commands that accept
> coding-system overrides.
>
>> Is there an equivalent of force_encoding('UTF-8') in Emacs?
>
> "C-x RET c utf-8 RET M-x SOME-COMMAND RET"

I see that 'C-x RET c' just sets coding-system-for-read and
coding-system-for-write for the next command, so could
base64-decode-region get coding from these variables?

> It will work if you encode "ä" first:
>
>   (decode-coding-string (base64-decode-string
>                          (base64-encode-string
> 			  (encode-coding-string "ä" 'utf-8)))
> 			'utf-8)

Thanks, this works for strings.

My real need was to find a way to decode base64 regions
that were encoded with UTF-8 coding.

First I tried to find such post-processing that would
recover "broken" characters inserted by base64-decode-region.
It seems these characters represent bytes that are parts
of the UTF-8 characters encoded in the UTF-8 buffer
using eight-bit charset.  I failed to find such functions
that would convert the result of base64-decode-region
to UTF-8 characters in the UTF-8 buffer.

So I wrote a replacement of base64-decode-region:

(defun base64-decode-utf8-region (beg end)
  (interactive "r")
  (replace-region-contents beg end
   (lambda ()
     (decode-coding-string
      (base64-decode-string
       (buffer-substring beg end))
      (or coding-system-for-write 'utf-8)))))

But the question remains: is it possible to do the same
in a simpler way without the need to write a new command?

bug#38587: base64-decode-region breaks encoding

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: schwab@linux-m68k.org,  larsi@gnus.org,  38587@debbugs.gnu.org
> Date: Mon, 16 Dec 2019 23:51:48 +0200
> 
> >> Is there an equivalent of force_encoding('UTF-8') in Emacs?
> >
> > "C-x RET c utf-8 RET M-x SOME-COMMAND RET"
> 
> I see that 'C-x RET c' just sets coding-system-for-read and
> coding-system-for-write for the next command, so could
> base64-decode-region get coding from these variables?

Yes, just access the variable and use the value.

> >   (decode-coding-string (base64-decode-string
> >                          (base64-encode-string
> > 			  (encode-coding-string "ä" 'utf-8)))
> > 			'utf-8)
> 
> Thanks, this works for strings.
> 
> My real need was to find a way to decode base64 regions
> that were encoded with UTF-8 coding.

Then you need just base64-decode-region followed by
decode-coding-region.  Assuming that I understand what you mean,
i.e. that the region you want to decode includes only ASCII characters
and raw bytes (otherwise it is not correct to say that it is "encoded
with UTF-8").

> First I tried to find such post-processing that would
> recover "broken" characters inserted by base64-decode-region.
> It seems these characters represent bytes that are parts
> of the UTF-8 characters encoded in the UTF-8 buffer
> using eight-bit charset.  I failed to find such functions
> that would convert the result of base64-decode-region
> to UTF-8 characters in the UTF-8 buffer.

decode-coding-region should be what you want.  It decodes raw bytes
(a.k.a. "eight-bit charset") into characters.

> So I wrote a replacement of base64-decode-region:
> 
> (defun base64-decode-utf8-region (beg end)
>   (interactive "r")
>   (replace-region-contents beg end
>    (lambda ()
>      (decode-coding-string
>       (base64-decode-string
>        (buffer-substring beg end))
>       (or coding-system-for-write 'utf-8)))))
> 
> But the question remains: is it possible to do the same
> in a simpler way without the need to write a new command?

Yes, see above.  In particular, decode-coding-region already knows how
to replace the region with the decoded text.

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

tags 38587 wontfix
close 38587 27.0.50
quit

> decode-coding-region should be what you want.  It decodes raw bytes
> (a.k.a. "eight-bit charset") into characters.

Thanks, I'm using this advice.

(advice-add 'base64-decode-region :after
            (lambda (beg end &optional _base64url)
              (decode-coding-region beg end buffer-file-coding-system))
            '((name . base64-decode-region-with-buffer-coding)))

So I'm closing this.  Not sure what could be added to documentation.

bug#38587: base64-decode-region breaks encoding

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> Thanks, I'm using this advice.
>
> (advice-add 'base64-decode-region :after
>             (lambda (beg end &optional _base64url)
>               (decode-coding-region beg end buffer-file-coding-system))
>             '((name . base64-decode-region-with-buffer-coding)))

I think in many cases this will work fine, but you probably will have
Emacs double-decode a lot of data, as the other code in Emacs will
normally call decode-coding-region (if you want to end up with text),
and you'll destroy the bits of your Emacs that decodes base64 data into
images and the like.

(If I read the advice correctly -- I'm not very familiar with advising
functions.)

> So I'm closing this.  Not sure what could be added to documentation.

Something equivalent to what the manual has to say about it would be
nice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38587: base64-decode-region breaks encoding

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Something equivalent to what the manual has to say about it would be
> nice.

I've now added a slightly vague version what the manual says here to the
doc strings, but I think it should be factually correct.

I also noticed that the encode-coding-region/decode-coding-region doc
strings do not actually say the most important thing -- whether
"encoding" goes from bytes to text or the other way around, so I've now
made this extremely explicit, because I think there's a lot of confusion
in the area, and "encode" and "decode" in themselves don't actually mean
anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38587: base64-decode-region breaks encoding

[Andreas Schwab]

On Dez 16 2019, Juri Linkov wrote:

> Also this doesn't work on the string output:
>
>   (decode-coding-string (base64-decode-string (base64-encode-string "ä"))
>                         'utf-8)

This works as expected, as the string to be decoded is not a valid UTF-8
sequence.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#38587: base64-decode-region breaks encoding

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> But is it still possible to tell base64-decode-region
> about the expected output coding system?

Anything is possible, but it doesn't make sense to complicate a function
like that in this manner.  These functions perform a transformation from
one set of octets to a different set of octets, and they have nothing to
do with characters.

We have a bunch of functions in Emacs that work on bytes, and not on
characters.  The way to use them is always (assuming you're starting
with something that is text) to use encode-coding-region first, and
(going in the opposite direction), if you want to end up with something
that is text afterwards, you have to call decode-coding-region
afterwards.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38587: base64-decode-region breaks encoding

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Sun, 15 Dec 2019 01:31:38 +0200
> Cc: 38587@debbugs.gnu.org
> 
> But when it's know that the source string was in UTF-8,
> shouldn't it be reversible?  What is needed for that?

The source string is not in UTF-8, it is in internal Emacs
representation of strings.

bug#38587: base64-decode-region breaks encoding

[Juri Linkov]

>> But when it's know that the source string was in UTF-8,
>> shouldn't it be reversible?  What is needed for that?
>
> The source string is not in UTF-8, it is in internal Emacs
> representation of strings.

Is internal Emacs representation compatible with UTF-8?

bug#38587: base64-decode-region breaks encoding

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: larsi@gnus.org,  38587@debbugs.gnu.org
> Date: Mon, 16 Dec 2019 00:41:48 +0200
> 
> >> But when it's know that the source string was in UTF-8,
> >> shouldn't it be reversible?  What is needed for that?
> >
> > The source string is not in UTF-8, it is in internal Emacs
> > representation of strings.
> 
> Is internal Emacs representation compatible with UTF-8?

It is (currently).  But it isn't UTF-8, strictly speaking.  In
particular, raw bytes are represented there by 2-byte sequences that
aren't valid UTF-8.