Re: emacs-25 b6b47af: Properly encode/decode base64Binary data in SOAP

Re: emacs-25 b6b47af: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Hi Andreas,

Andreas Schwab <schwab@gnu.org> writes:

> branch: emacs-25
> commit b6b47af82f6c7d960388ec46dd8ab371c2e34de4
> Author: Andreas Schwab <schwab@linux-m68k.org>
> Commit: Andreas Schwab <schwab@linux-m68k.org>
>
>     Properly encode/decode base64Binary data in SOAP
>     
>     	* lisp/net/soap-client.el (soap-encode-xs-basic-type): Encode
>     	base64Binary value as utf-8.
>     	(soap-decode-xs-basic-type): Decode base64Binary value as utf-8.
> ---
>  lisp/net/soap-client.el |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lisp/net/soap-client.el b/lisp/net/soap-client.el
> index f8cdaa9..7402464 100644
> --- a/lisp/net/soap-client.el
> +++ b/lisp/net/soap-client.el
> @@ -538,7 +538,7 @@ This is a specialization of `soap-encode-value' for
>                 (base64Binary
>                  (unless (stringp value)
>                    (error "Not a string value for base64Binary"))
> -                (base64-encode-string value))
> +                (base64-encode-string (encode-coding-string value 'utf-8)))
>  
>                 (otherwise
>                  (error "Don't know how to encode %s for type %s"
> @@ -682,7 +682,7 @@ This is a specialization of `soap-decode-type' for
>                 decimal byte float double duration)
>           (string-to-number (car contents)))
>          (boolean (string= (downcase (car contents)) "true"))
> -        (base64Binary (base64-decode-string (car contents)))
> +        (base64Binary (decode-coding-string (base64-decode-string (car contents)) 'utf-8))
>          (anyType (soap-decode-any-type node))
>          (Array (soap-decode-array node))))))
>  

I'm trying to merge Emacs master to the emacs-soap-client repository
(and back) to release soap-client 3.1.0.  This patch is causing some
test suite failures.  (Unfortunately the test suite is private because
we're not sure if we can redistribute the WSDL files therein.)  Here is
an example fragment that fails with your patch:

<originator xsi:type="xsd:base64Binary">Wm9sdMOhbiBLcmFqY3NvdmljcyA8enVsdGhhbmtAZ21haWwuY29tPg==</originator>

The test suite expects this to decode as:

Zolt\303\241n Krajcsovics [...]

but gets:

Zoltán Krajcsovics [...]

base64Binary is supposed to be for binary data; I don't think it's safe
to assume the content is a UTF-8-encoded string for encoding or
decoding, in general.

Do you have an example SOAP message that your patch fixes, and can the
surrounding code do the UTF-8 encoding/decoding instead of soap-client
itself?

Thanks,
Thomas

Re: emacs-25 b6b47af: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> <originator xsi:type="xsd:base64Binary">Wm9sdMOhbiBLcmFqY3NvdmljcyA8enVsdGhhbmtAZ21haWwuY29tPg==</originator>
>
> The test suite expects this to decode as:
>
> Zolt\303\241n Krajcsovics [...]

Leaving it undecoded is definitely wrong.  Undecoded strings should
never be used inside Emacs, except for interaction with external
sources.

> Do you have an example SOAP message that your patch fixes, and can the
> surrounding code do the UTF-8 encoding/decoding instead of soap-client
> itself?

(debbugs-get-status 22285)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> <originator xsi:type="xsd:base64Binary">Wm9sdMOhbiBLcmFqY3NvdmljcyA8enVsdGhhbmtAZ21haWwuY29tPg==</originator>
>>
>> The test suite expects this to decode as:
>>
>> Zolt\303\241n Krajcsovics [...]
>
> Leaving it undecoded is definitely wrong.  Undecoded strings should
> never be used inside Emacs, except for interaction with external
> sources.

Agreed, but I disagree that soap-client itself should be doing the
decoding.  SOAP messages are sent to and received from external sources.
In general, an elisp function receiving base64-encoded data from an
external source, via a soap-client call, will want to see it first as a
unibyte string of bytes, in case it's image data or some other
non-string data.  If the soap-client caller expects it to be a UTF-8
encoded string, the caller can attempt the string decoding.

>> Do you have an example SOAP message that your patch fixes, and can the
>> surrounding code do the UTF-8 encoding/decoding instead of soap-client
>> itself?
>
> (debbugs-get-status 22285)

OK, thanks.  The originator field there is base64 binary.  Without your
patch:

(multibyte-string-p (cdr (assoc 'originator (car (debbugs-get-status 22285)))))
=> nil
(length (cdr (assq 'originator (car (debbugs-get-status 22285)))))
=> 51

With your patch:

(multibyte-string-p (cdr (assoc 'originator (car (debbugs-get-status 22285)))))
=> t
(length (cdr (assq 'originator (car (debbugs-get-status 22285)))))
=> 50

Before your patch, what problem did you see?  Did some caller of
debbugs-get-status show raw characters somewhere, i.e., did it assume
originator was an already-decoded UTF-8 string?  If so, I think that
caller should be fixed (and soap-client should go back to its old
behavior of leaving the string undecoded).

To see what other callers do, I tried (debbugs-org-bugs 22285); it
decodes the originator field itself:

  [...]
  (originator (when (cdr (assq 'originator status))
                (decode-coding-string
                 (cdr (assq 'originator status)) 'utf-8)))
  [...]

which correctly shows the "LATIN SMALL LETTER E WITH ACUTE" in the
*Org Bugs* buffer with the old soap-client behavior.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Before your patch, what problem did you see?

An undecoded string.

> To see what other callers do, I tried (debbugs-org-bugs 22285); it
> decodes the originator field itself:
>
>   [...]
>   (originator (when (cdr (assq 'originator status))
>                 (decode-coding-string
>                  (cdr (assq 'originator status)) 'utf-8)))

I expect that to be done at the transport level.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Alex Harsanyi]

2016-03-11 16:35 GMT+08:00 Andreas Schwab <schwab@linux-m68k.org>:
> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> Before your patch, what problem did you see?
>
> An undecoded string.
>
>> To see what other callers do, I tried (debbugs-org-bugs 22285); it
>> decodes the originator field itself:
>>
>>   [...]
>>   (originator (when (cdr (assq 'originator status))
>>                 (decode-coding-string
>>                  (cdr (assq 'originator status)) 'utf-8)))
>
> I expect that to be done at the transport level.

Unfortunately, there is not enough information to decode the string at
the transport level.  What if the base64 data contains a PNG image or
some other binary data?

Best Regards,
Alex.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Alex Harsanyi <alexharsanyi@gmail.com> writes:

> Unfortunately, there is not enough information to decode the string at
> the transport level.

Then provide an interface to decode it.  In Emacs you always see the
decoded contents unless you explicitly ask for not doing it.  Returning
half-decoded content is the wrong way to do it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> Then provide an interface to decode it.  In Emacs you always see the
> decoded contents unless you explicitly ask for not doing it.  Returning
> half-decoded content is the wrong way to do it.

AFAIK it's not half-decoded, it's just never decoded because the
transport format only deals with bytes.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Then provide an interface to decode it.  In Emacs you always see the
>> decoded contents unless you explicitly ask for not doing it.  Returning
>> half-decoded content is the wrong way to do it.
>
> AFAIK it's not half-decoded, it's just never decoded because the
> transport format only deals with bytes.

The interface does not return bytes, it returns a (half-)decoded lisp
form.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> The interface does not return bytes, it returns a (half-)decoded lisp
> form.

My understanding is that it returns a structure whose leaves are
all byte-strings because.

In this sense, they're all "undecoded".  And the meaning of those
byte-strings is not specified by the generic format, so the generic
parser can't know if or how to decode them.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The interface does not return bytes, it returns a (half-)decoded lisp
>> form.
>
> My understanding is that it returns a structure whose leaves are
> all byte-strings because.

String values are returned as multibyte, e.g.:

(multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
=> t

because parsing happens in a temporary buffer where
enable-multibyte-characters is set, and the string value is returned
unchanged.

> In this sense, they're all "undecoded".  And the meaning of those
> byte-strings is not specified by the generic format, so the generic
> parser can't know if or how to decode them.

This is true for base64Binary values.

I'd like to change the base64Binary behavior back, but we can document
this nuance of the API.  Also, on the encoding side I think we should
force the caller to provide unibyte strings for base64Binary values.

Is the attached patch OK for master and emacs-25?

Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-soap-client-document-base64-handling.patch --]
[-- Type: text/x-patch, Size: 1824 bytes --]

diff --git a/lisp/net/soap-client.el b/lisp/net/soap-client.el
index 7402464..bee3a12 100644
--- a/lisp/net/soap-client.el
+++ b/lisp/net/soap-client.el
@@ -536,9 +536,10 @@ soap-encode-xs-basic-type
                 (number-to-string value))
 
                (base64Binary
-                (unless (stringp value)
-                  (error "Not a string value for base64Binary"))
-                (base64-encode-string (encode-coding-string value 'utf-8)))
+                (unless (and (stringp value)
+                             (not (multibyte-string-p value)))
+                  (error "Not a unibyte string value for base64Binary"))
+                (base64-encode-string value))
 
                (otherwise
                 (error "Don't know how to encode %s for type %s"
@@ -682,7 +683,7 @@ soap-decode-xs-basic-type
                decimal byte float double duration)
          (string-to-number (car contents)))
         (boolean (string= (downcase (car contents)) "true"))
-        (base64Binary (decode-coding-string (base64-decode-string (car contents)) 'utf-8))
+        (base64Binary (base64-decode-string (car contents)))
         (anyType (soap-decode-any-type node))
         (Array (soap-decode-array node))))))
 
@@ -3096,7 +3097,9 @@ soap-invoke
 NOTE: The SOAP service provider should document the available
 operations and their parameters for the service.  You can also
 use the `soap-inspect' function to browse the available
-operations in a WSDL document."
+operations in a WSDL document.  `soap-invoke' base64-decodes
+base64Binary return values into unibyte strings; these
+byte-strings require further interpretation by the caller."
   (apply #'soap-invoke-internal nil nil wsdl service operation-name parameters))
 
 (defun soap-invoke-async (callback cbargs wsdl service operation-name

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

>>> The interface does not return bytes, it returns a (half-)decoded lisp
>>> form.
>> My understanding is that it returns a structure whose leaves are
>> all byte-strings because.
> String values are returned as multibyte, e.g.:
> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
> => t
> because parsing happens in a temporary buffer where
> enable-multibyte-characters is set, and the string value is returned
> unchanged.

If these are undecoded, then I'd consider it a bug to return them as
multibyte, indeed.  The temp buffer should probably be made unibyte.

> this nuance of the API.  Also, on the encoding side I think we should
> force the caller to provide unibyte strings for base64Binary values.

I'd let base64-encode-string make that decision, but it's just my own
preference of bikeshed's color.

> Is the attached patch OK for master and emacs-25?

Looks OK to me for master.  I'll let others figure out if it's safe
enough for `emacs-25'.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> The interface does not return bytes, it returns a (half-)decoded lisp
>>>> form.
>>> My understanding is that it returns a structure whose leaves are
>>> all byte-strings because.
>> String values are returned as multibyte, e.g.:
>> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
>> => t
>> because parsing happens in a temporary buffer where
>> enable-multibyte-characters is set, and the string value is returned
>> unchanged.
>
> If these are undecoded, then I'd consider it a bug to return them as
> multibyte, indeed.  The temp buffer should probably be made unibyte.

See my response to Eli.

>> this nuance of the API.  Also, on the encoding side I think we should
>> force the caller to provide unibyte strings for base64Binary values.
>
> I'd let base64-encode-string make that decision, but it's just my own
> preference of bikeshed's color.

OK.

>> Is the attached patch OK for master and emacs-25?
>
> Looks OK to me for master.  I'll let others figure out if it's safe
> enough for `emacs-25'.

Andreas committed his change to base64Binary handling to both master and
emacs-25.  Whatever the decision regarding reverting it, those two
branches should have the same behavior.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Date: Sat, 12 Mar 2016 22:52:25 -0500
> Cc: emacs-devel@gnu.org
> 
> String values are returned as multibyte, e.g.:
> 
> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
> => t

They should be unibyte, I agree with Stefan.  If you return multibyte
strings, they should be decoded.

> Is the attached patch OK for master and emacs-25?

Doesn't it bring back the bug which caused Andreas to make the change
you want to undo?

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Date: Sat, 12 Mar 2016 22:52:25 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> String values are returned as multibyte, e.g.:
>> 
>> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
>> => t
>> because parsing happens in a temporary buffer where
>> enable-multibyte-characters is set, and the string value is returned
>> unchanged.
>
> They should be unibyte, I agree with Stefan.  If you return multibyte
> strings, they should be decoded.

Sorry, I wasn't being precise when I said "unchanged".  The result from
the server goes through several steps before soap-client returns the
multibyte string result:

   (defun soap-parse-server-response ()
     "Error-check and parse the XML contents of the current buffer."
     (let ((mime-part (mm-dissect-buffer t t)))
       (unless mime-part
         (error "Failed to decode response from server"))
       (unless (equal (car (mm-handle-type mime-part)) "text/xml")
         (error "Server response is not an XML document"))
       (with-temp-buffer
         (mm-insert-part mime-part)
         (prog1
             (car (xml-parse-region (point-min) (point-max)))
           (kill-buffer)
           (mm-destroy-part mime-part)))))

mm-insert-part does:

   (string-to-multibyte (mm-get-part handle no-cache))

In cases where the caller is expecting an xsd:string, the idea is for
soap-client to return a native Emacs string, for the caller's
convenience.  I guess soap-client assumes that the mm and xml packages
will do the right thing to convert XML string values into Emacs's
internal format.

>> Is the attached patch OK for master and emacs-25?
>
> Doesn't it bring back the bug which caused Andreas to make the change
> you want to undo?

It brings back the behavior of soap-client returning base64-decoded
xsd:base64Binary values as unibyte strings.  The debate on this thread
is about whether that behavior is buggy or not.  But yes, I want to
revert Andreas's change on both master and emacs-25 branches, because I
don't consider the old behavior buggy.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 13 Mar 2016 13:57:32 -0400
> 
>    (defun soap-parse-server-response ()
>      "Error-check and parse the XML contents of the current buffer."
>      (let ((mime-part (mm-dissect-buffer t t)))
>        (unless mime-part
>          (error "Failed to decode response from server"))
>        (unless (equal (car (mm-handle-type mime-part)) "text/xml")
>          (error "Server response is not an XML document"))
>        (with-temp-buffer
>          (mm-insert-part mime-part)
>          (prog1
>              (car (xml-parse-region (point-min) (point-max)))
>            (kill-buffer)
>            (mm-destroy-part mime-part)))))
> 
> mm-insert-part does:
> 
>    (string-to-multibyte (mm-get-part handle no-cache))

Why does it do that?  string-to-multibyte is one of those functions
that should never be used.

> In cases where the caller is expecting an xsd:string, the idea is for
> soap-client to return a native Emacs string, for the caller's
> convenience.

But that's not what string-to-multibyte does.

> I guess soap-client assumes that the mm and xml packages will do the
> right thing to convert XML string values into Emacs's internal
> format.

I'm not sure we are not mis-communicating: conversion into internal
format is what decoding does.  Whereas you just said a few messages
upthread that you thought strings should be returned undecoded,
i.e. as binary streams of bytes.  What am I missing?

> >> Is the attached patch OK for master and emacs-25?
> >
> > Doesn't it bring back the bug which caused Andreas to make the change
> > you want to undo?
> 
> It brings back the behavior of soap-client returning base64-decoded
> xsd:base64Binary values as unibyte strings.

I'm confused: you've just demonstrated that it returns them as
multibyte strings with raw bytes in their multibyte encoding.

> The debate on this thread is about whether that behavior is buggy or
> not.  But yes, I want to revert Andreas's change on both master and
> emacs-25 branches, because I don't consider the old behavior buggy.

That'll bring the bug in the debbugs package back, I think.  Once
again, if you want to return undecoded strings, they should at the
very least be unibyte, not multibyte.  Apologies if I'm too confused
to talk intelligently about this.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 13 Mar 2016 13:57:32 -0400
>> 
>>    (defun soap-parse-server-response ()
>>      "Error-check and parse the XML contents of the current buffer."
>>      (let ((mime-part (mm-dissect-buffer t t)))
>>        (unless mime-part
>>          (error "Failed to decode response from server"))
>>        (unless (equal (car (mm-handle-type mime-part)) "text/xml")
>>          (error "Server response is not an XML document"))
>>        (with-temp-buffer
>>          (mm-insert-part mime-part)
>>          (prog1
>>              (car (xml-parse-region (point-min) (point-max)))
>>            (kill-buffer)
>>            (mm-destroy-part mime-part)))))
>> 
>> mm-insert-part does:
>> 
>>    (string-to-multibyte (mm-get-part handle no-cache))
>
> Why does it do that?  string-to-multibyte is one of those functions
> that should never be used.

I don't know.  This is the first I've looked at the mm code.  I'll have
to do more investigation here, apparently.

>> In cases where the caller is expecting an xsd:string, the idea is for
>> soap-client to return a native Emacs string, for the caller's
>> convenience.
>
> But that's not what string-to-multibyte does.
>
>> I guess soap-client assumes that the mm and xml packages will do the
>> right thing to convert XML string values into Emacs's internal
>> format.
>
> I'm not sure we are not mis-communicating: conversion into internal
> format is what decoding does.  Whereas you just said a few messages
> upthread that you thought strings should be returned undecoded,
> i.e. as binary streams of bytes.  What am I missing?

The discussion expanded from being about xsd:base64Binary, to being
about all strings returned by soap-client (see below).  Upthread I was
saying only that xsd:base64Binary values should be returned undecoded.
I wasn't commenting on how other XSD string values (xsd:string, etc.)
should be returned.

>> >> Is the attached patch OK for master and emacs-25?
>> >
>> > Doesn't it bring back the bug which caused Andreas to make the change
>> > you want to undo?
>> 
>> It brings back the behavior of soap-client returning base64-decoded
>> xsd:base64Binary values as unibyte strings.
>
> I'm confused: you've just demonstrated that it returns them as
> multibyte strings with raw bytes in their multibyte encoding.
>
>> The debate on this thread is about whether that behavior is buggy or
>> not.  But yes, I want to revert Andreas's change on both master and
>> emacs-25 branches, because I don't consider the old behavior buggy.
>
> That'll bring the bug in the debbugs package back, I think.  Once
> again, if you want to return undecoded strings, they should at the
> very least be unibyte, not multibyte.  Apologies if I'm too confused
> to talk intelligently about this.

Apologies for helping lead to confusion; it's good to have you reviewing
soap-client's design.

The discussion expanded from being about how to handle xsd:base64Binary
values only (Andreas's patch), to about how soap-client handles all
strings (including xsd:string, etc.).  It could be that how soap-client
handles all strings is broken, since it appears to be relying on
string-to-multibyte which you're saying should never be used.  However,
soap-client's decoding has been good enough that no one has complained
about string handling in general up til now.  But I'll review the design
with Alex to see if we can avoid calling string-to-multibyte via mm.

Maybe I can give an example with XML fragments returned by the server,
to show how I think soap-client should handle xsd:base64Binary values.

The debbugs server will respond with:

<?xml version="1.0" encoding="UTF-8"?>
[...]
<severity xsi:type="xsd:string">normal</severity>
[...]
<originator xsi:type="xsd:base64Binary">Q2zDqW1lbnQgUGl0LS1DbGF1ZGVsIDxjbGVtZW50LnBpdGNsYXVkZWxAbGl2ZS5jb20+</originator>
[...]

soap-client will parse those results into a structure that it returns to
the caller:

([...]
 (severity . "<string1>")
 [...]
 (originator . "<string2>")
 [...])

I think <string2> should be unibyte, because xsd:base64Binary represents
binary data, not necessarily a string.  It was unibyte before Andreas's
patch.  His patch changed it to be multibyte, by assuming the binary
data is a UTF-8 string and decoding it into Emacs's internal format.

What <string1> should be (unibyte or multibyte) and how it should be
produced (decoded) is the broader discussion.  I don't know enough to
have an opinion on that yet, other than it seems to have been working to
treat it as multibyte up until now.  Again, I'll have to talk to Alex
about this.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 13 Mar 2016 15:54:34 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> >> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Sun, 13 Mar 2016 13:57:32 -0400
> >> 
> >>    (defun soap-parse-server-response ()
> >>      "Error-check and parse the XML contents of the current buffer."
> >>      (let ((mime-part (mm-dissect-buffer t t)))
> >>        (unless mime-part
> >>          (error "Failed to decode response from server"))
> >>        (unless (equal (car (mm-handle-type mime-part)) "text/xml")
> >>          (error "Server response is not an XML document"))
> >>        (with-temp-buffer
> >>          (mm-insert-part mime-part)
> >>          (prog1
> >>              (car (xml-parse-region (point-min) (point-max)))
> >>            (kill-buffer)
> >>            (mm-destroy-part mime-part)))))
> >> 
> >> mm-insert-part does:
> >> 
> >>    (string-to-multibyte (mm-get-part handle no-cache))
> >
> > Why does it do that?  string-to-multibyte is one of those functions
> > that should never be used.
> 
> I don't know.  This is the first I've looked at the mm code.  I'll have
> to do more investigation here, apparently.

IME, mm-decode has a lot of baggage from distant past, when it had to
deal with bugs in Emacs, with incompatibilities between Emacs and
XEmacs, and from its own misconceptions in the area of encoding and
decoding text.  Its code should be carefully reviewed for correctness.

> Upthread I was saying only that xsd:base64Binary values should be
> returned undecoded.

But it currently doesn't, AFAIU, since this:

  (cdr (assq 'severity (car (debbugs-get-status 22285))))

is returned as a multibyte string.  So I guess you are saying that
soap-client needs some changes to return xsd:base64Binary data as
unibyte strings?  Or is "severity" an xsd:string?

> I wasn't commenting on how other XSD string values (xsd:string, etc.)
> should be returned.

In general, if it's known to be a text string, and its encoding is
specified by the document, or can be deduced otherwise with a 100%
certainty, I'd recommend to return decoded strings.

> Maybe I can give an example with XML fragments returned by the server,
> to show how I think soap-client should handle xsd:base64Binary values.
> 
> The debbugs server will respond with:
> 
> <?xml version="1.0" encoding="UTF-8"?>
> [...]
> <severity xsi:type="xsd:string">normal</severity>
> [...]
> <originator xsi:type="xsd:base64Binary">Q2zDqW1lbnQgUGl0LS1DbGF1ZGVsIDxjbGVtZW50LnBpdGNsYXVkZWxAbGl2ZS5jb20+</originator>
> [...]
> 
> soap-client will parse those results into a structure that it returns to
> the caller:
> 
> ([...]
>  (severity . "<string1>")
>  [...]
>  (originator . "<string2>")
>  [...])
> 
> I think <string2> should be unibyte, because xsd:base64Binary represents
> binary data, not necessarily a string.

Btw, why is "originator" not a string? why xsd:base64Binary?  It's a
name of a human (or some other entity), so it's clearly text, no?

> What <string1> should be (unibyte or multibyte) and how it should be
> produced (decoded) is the broader discussion.  I don't know enough to
> have an opinion on that yet, other than it seems to have been working to
> treat it as multibyte up until now.  Again, I'll have to talk to Alex
> about this.

If you can reliably decode it, then multibyte and decoded is better.
I'd also say that if it's known that xsd:base64Binary is a string in
disguise, it should also be decoded.  IOW, whenever the data is a
string, it is better to decode it, I agree with Andreas here: unibyte
strings that represent text are a PITA without a good justification.

Thanks.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 13 Mar 2016 15:54:34 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> >> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> >> Date: Sun, 13 Mar 2016 13:57:32 -0400
>> >> 
>> >>    (defun soap-parse-server-response ()
>> >>      "Error-check and parse the XML contents of the current buffer."
>> >>      (let ((mime-part (mm-dissect-buffer t t)))
>> >>        (unless mime-part
>> >>          (error "Failed to decode response from server"))
>> >>        (unless (equal (car (mm-handle-type mime-part)) "text/xml")
>> >>          (error "Server response is not an XML document"))
>> >>        (with-temp-buffer
>> >>          (mm-insert-part mime-part)
>> >>          (prog1
>> >>              (car (xml-parse-region (point-min) (point-max)))
>> >>            (kill-buffer)
>> >>            (mm-destroy-part mime-part)))))
>> >> 
>> >> mm-insert-part does:
>> >> 
>> >>    (string-to-multibyte (mm-get-part handle no-cache))
>> >
>> > Why does it do that?  string-to-multibyte is one of those functions
>> > that should never be used.
>> 
>> I don't know.  This is the first I've looked at the mm code.  I'll have
>> to do more investigation here, apparently.
>
> IME, mm-decode has a lot of baggage from distant past, when it had to
> deal with bugs in Emacs, with incompatibilities between Emacs and
> XEmacs, and from its own misconceptions in the area of encoding and
> decoding text.  Its code should be carefully reviewed for correctness.

OK.

>> Upthread I was saying only that xsd:base64Binary values should be
>> returned undecoded.
>
> But it currently doesn't, AFAIU, since this:
>
>   (cdr (assq 'severity (car (debbugs-get-status 22285))))
>
> is returned as a multibyte string.

"severity" is an xsd:string.

> So I guess you are saying that soap-client needs some changes to
> return xsd:base64Binary data as unibyte strings?

Yes, because Andreas's patch changed the behavior.  But the "severity"
example doesn't demonstrate this.  This does, because originator is an
xsd:base64Binary:

(cdr (assq 'originator (car (debbugs-get-status 22285))))

Before Andreas's patch that was unibyte, now, with Andreas's patch, it's
multibyte.

> Or is "severity" an xsd:string?

Yes.

>> I wasn't commenting on how other XSD string values (xsd:string, etc.)
>> should be returned.
>
> In general, if it's known to be a text string, and its encoding is
> specified by the document, or can be deduced otherwise with a 100%
> certainty, I'd recommend to return decoded strings.

OK.  That's what soap-client currently tries to do.  But as you've
pointed out, we need to review what it does for correctness.

>> Maybe I can give an example with XML fragments returned by the server,
>> to show how I think soap-client should handle xsd:base64Binary values.
>> 
>> The debbugs server will respond with:
>> 
>> <?xml version="1.0" encoding="UTF-8"?>
>> [...]
>> <severity xsi:type="xsd:string">normal</severity>
>> [...]
>> <originator xsi:type="xsd:base64Binary">Q2zDqW1lbnQgUGl0LS1DbGF1ZGVsIDxjbGVtZW50LnBpdGNsYXVkZWxAbGl2ZS5jb20+</originator>
>> [...]
>> 
>> soap-client will parse those results into a structure that it returns to
>> the caller:
>> 
>> ([...]
>>  (severity . "<string1>")
>>  [...]
>>  (originator . "<string2>")
>>  [...])
>> 
>> I think <string2> should be unibyte, because xsd:base64Binary represents
>> binary data, not necessarily a string.
>
> Btw, why is "originator" not a string? why xsd:base64Binary?  It's a
> name of a human (or some other entity), so it's clearly text, no?

Good question, I don't know.  Maybe Michael could comment here, since
this was a Debbugs decision.

>> What <string1> should be (unibyte or multibyte) and how it should be
>> produced (decoded) is the broader discussion.  I don't know enough to
>> have an opinion on that yet, other than it seems to have been working to
>> treat it as multibyte up until now.  Again, I'll have to talk to Alex
>> about this.
>
> If you can reliably decode it, then multibyte and decoded is better.
> I'd also say that if it's known that xsd:base64Binary is a string in
> disguise, it should also be decoded.

OK.  But there is no way for soap-client itself to know what the content
of an xsd:base64Binary value is.  The caller of soap-client will know
what it is, and so it should be up to the caller to interpret the bytes.
That's what Debbugs does with "originator"; it decodes them assuming
they represent a UTF-8 string.

> IOW, whenever the data is a string, it is better to decode it, I agree
> with Andreas here: unibyte strings that represent text are a PITA
> without a good justification.

I agree with you and Andreas in general.  However an xsd:base64Binary
value is a special case because it does not always represent text.
Therefore, I disagree with Andreas's patch; I've been trying to provide
good justification for not assuming that xsd:base64Binary values
represent text, because they don't always represent text, and there's no
way for soap-client to know when they do.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 13 Mar 2016 17:17:04 -0400
> 
> I agree with you and Andreas in general.  However an xsd:base64Binary
> value is a special case because it does not always represent text.
> Therefore, I disagree with Andreas's patch; I've been trying to provide
> good justification for not assuming that xsd:base64Binary values
> represent text, because they don't always represent text, and there's no
> way for soap-client to know when they do.

Andreas, is it possible to move the decoding of originator from
soap-client to debbugs?  Or were there other problems you found, in
addition to decoding the originator?

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Andreas, is it possible to move the decoding of originator from
> soap-client to debbugs?

That needs to be answered by its author.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Andreas Schwab <schwab@suse.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Andreas, is it possible to move the decoding of originator from
>> soap-client to debbugs?
>
> That needs to be answered by its author.

As explained in my other message, originator (and other attributes)
could be sent by the debbugs server as either xsd:string or
xsd:base64Binary. debbugs.el does not know which encoding has been
applied, it must trust on consistent soap-client.el decoding.

> Andreas.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> As explained in my other message, originator (and other attributes)
> could be sent by the debbugs server as either xsd:string or
> xsd:base64Binary.

Why?  Doesn't the schema specify these constraints?
I mean, isn't this an issue that should be solved between the
soap-client client (i.e. debbugs.el) and the SOAP server?

And of course, in case both are possible for historical reasons and the
debbugs.el isn't told which version of the schema is used, debbugs.el
can check the multibyteness of the string.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Alex Harsanyi]

2016-03-14 19:56 GMT+08:00 Stefan Monnier <monnier@iro.umontreal.ca>:
>> As explained in my other message, originator (and other attributes)
>> could be sent by the debbugs server as either xsd:string or
>> xsd:base64Binary.
>
> Why?  Doesn't the schema specify these constraints?

The debuggs schema (WSDL document) specifies that parameters can be of
"any" type.  This means that the server encodes type information with
each parameter (the "xsi:type" attributes) and soap-client will use
this info to decode the contents of the XML tag into lisp data.

Unfortunately, the Perl SOAP sever works without a schema (it will
simply encode a list of values into an XML document based on their
type).  Since the SOAP server does not read the WSDL schema, changing
it will not help us.

> I mean, isn't this an issue that should be solved between the
> soap-client client (i.e. debbugs.el) and the SOAP server?

Yes this is what Thomas and me are arguing for :-)

Best Regards,
Alex.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Alex Harsanyi]

2016-03-14 17:15 GMT+08:00 Michael Albinus <michael.albinus@gmx.de>:
> Andreas Schwab <schwab@suse.de> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Andreas, is it possible to move the decoding of originator from
>>> soap-client to debbugs?
>>
>> That needs to be answered by its author.
>
> As explained in my other message, originator (and other attributes)
> could be sent by the debbugs server as either xsd:string or
> xsd:base64Binary. debbugs.el does not know which encoding has been
> applied, it must trust on consistent soap-client.el decoding.

The decoding in soap-client is consistent, even though not in the way
you would like it :-)

  * if it is told that a value is a string, it will return a string,
  * if it is told that a value is a byte array it will return a byte
array (unibyte string)

To use another example, if the SOAP server chooses to send all
parameters as strings:

    <example xsi:type="xsd:string">1234</example>

soap-client will decode them as strings, even though to the human
reader it is "obvious" that it is a number:

   '(example . "1234")

I think the problem here is that the debbugs server encodes utf8
values as base64, even though the message envelope is utf8 XML and
could handle them as normal strings.   debbugs.el does not want to
know about this, so expects strings to be strings.  soap-client.el is
caught in the middle.

I would also like to reiterate that base64 encoding can be used for
other things, such as images, and it would not be appropriate to
decode those as utf8 (not to mention that such a decoding might fail).

Best Regards,
Alex.

>
>> Andreas.
>
> Best regards, Michael.
>

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Alex Harsanyi <alexharsanyi@gmail.com> writes:

> I think the problem here is that the debbugs server encodes utf8
> values as base64, even though the message envelope is utf8 XML and
> could handle them as normal strings.   debbugs.el does not want to
> know about this, so expects strings to be strings.  soap-client.el is
> caught in the middle.

Well, if debbugs.el would get an indication from soap-client.el, whether
a string was encoded as xsd:string or xsd:base64Binary, it could decode
the latter values itself. Less convenient, but so what.

> I would also like to reiterate that base64 encoding can be used for
> other things, such as images, and it would not be appropriate to
> decode those as utf8 (not to mention that such a decoding might fail).

Is there a way to tell soap-client.el, what to do with base64 encoded values?
Something like a user-defined function, or alike?

> Best Regards,
> Alex.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Alex Harsanyi]

2016-03-14 20:38 GMT+08:00 Michael Albinus <michael.albinus@gmx.de>:
> Alex Harsanyi <alexharsanyi@gmail.com> writes:
>
>> I think the problem here is that the debbugs server encodes utf8
>> values as base64, even though the message envelope is utf8 XML and
>> could handle them as normal strings.   debbugs.el does not want to
>> know about this, so expects strings to be strings.  soap-client.el is
>> caught in the middle.
>
> Well, if debbugs.el would get an indication from soap-client.el, whether
> a string was encoded as xsd:string or xsd:base64Binary, it could decode
> the latter values itself. Less convenient, but so what.

We were discussing with Thomas to have soap-client.el return the
base64 string "as is" and let the caller decode and process it.
Unfortunately, there is no base64-string-p function and this value
would look like a (multibyte) string.  Perhaps we could return (cons
'base64 value), or a more appropriate type for a "array of bytes"
concept.  I'm open to suggestions.

>
>> I would also like to reiterate that base64 encoding can be used for
>> other things, such as images, and it would not be appropriate to
>> decode those as utf8 (not to mention that such a decoding might fail).
>
> Is there a way to tell soap-client.el, what to do with base64 encoded values?
> Something like a user-defined function, or alike?

This would not work in the general case except when all base64 encoded
values are the same underlying type (like utf8).

Although, given that debbugs.el is the only soap-client client that is
affected by this, we can add a simple "soap-base64-handler" function
to do what needs doing, and we can always extend that interface later
as more use cases emerge.

Best Regards,
Alex.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Alex Harsanyi <alexharsanyi@gmail.com> writes:

> We were discussing with Thomas to have soap-client.el return the
> base64 string "as is" and let the caller decode and process it.
> Unfortunately, there is no base64-string-p function and this value
> would look like a (multibyte) string.  Perhaps we could return (cons
> 'base64 value), or a more appropriate type for a "array of bytes"
> concept.  I'm open to suggestions.

Array of bytes sounds appropriate to me. This is similar to D-Bus, which
uses an array of bytes in case it cannot marshal the data into a proper type.

>>> I would also like to reiterate that base64 encoding can be used for
>>> other things, such as images, and it would not be appropriate to
>>> decode those as utf8 (not to mention that such a decoding might fail).
>>
>> Is there a way to tell soap-client.el, what to do with base64 encoded values?
>> Something like a user-defined function, or alike?
>
> This would not work in the general case except when all base64 encoded
> values are the same underlying type (like utf8).

Your default could behave as you like. It would be the responsibility of
the calling library to replace the default by a proper
function. debbugs.el would know, that all base64 encoded data must be
utf8.

> Although, given that debbugs.el is the only soap-client client that is
> affected by this, we can add a simple "soap-base64-handler" function
> to do what needs doing, and we can always extend that interface later
> as more use cases emerge.

Would be OK to me also.

> Best Regards,
> Alex.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> Well, if debbugs.el would get an indication from soap-client.el, whether
> a string was encoded as xsd:string or xsd:base64Binary, it could decode
> the latter values itself.

The multibyteness of the string should be exactly that indication.

> Less convenient, but so what.

Agree, but I don't think soap-client.el can do much more.  It's due to
a problem on the server side.

> Is there a way to tell soap-client.el, what to do with base64 encoded values?

Just a note: "base64" here might be a convenient word to use, but it
focuses on the wrong part of the issue.  soap-client already does the
base64 decoding.
The issue is that the server sends "base64Binary" where the important
word is "Binary".


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Well, if debbugs.el would get an indication from soap-client.el, whether
>> a string was encoded as xsd:string or xsd:base64Binary, it could decode
>> the latter values itself.
>
> The multibyteness of the string should be exactly that indication.

It isn't, at least in its current implementation. Let's take our example
(debbugs-get-status 22285). The debbugs server returns (beside other
data)

<severity xsi:type="xsd:string">normal</severity>
<originator xsi:type="xsd:base64Binary">Q2zDqW1lbnQgUGl0LS1DbGF1ZGVsIDxjbGVtZW50LnBpdGNsYXVkZWxAbGl2ZS5jb20+</originator>

Testing multibyteness returns

(multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
t
(multibyte-string-p (cdr (assq 'originator (car (debbugs-get-status 22285)))))
t

>         Stefan

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Michael Albinus <michael.albinus@gmx.de> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> Well, if debbugs.el would get an indication from soap-client.el, whether
>>> a string was encoded as xsd:string or xsd:base64Binary, it could decode
>>> the latter values itself.
>>
>> The multibyteness of the string should be exactly that indication.
>
> It isn't, at least in its current implementation. Let's take our example
> (debbugs-get-status 22285). The debbugs server returns (beside other
> data)
>
> <severity xsi:type="xsd:string">normal</severity>
> <originator xsi:type="xsd:base64Binary">Q2zDqW1lbnQgUGl0LS1DbGF1ZGVsIDxjbGVtZW50LnBpdGNsYXVkZWxAbGl2ZS5jb20+</originator>
>
> Testing multibyteness returns
>
> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
> t
> (multibyte-string-p (cdr (assq 'originator (car (debbugs-get-status 22285)))))
> t

Yes, that's because of Andreas's soap-client patch
(b6b47af82f6c7d960388ec46dd8ab371c2e34de4), the patch under discussion
that I'd like to revert.  Without that patch, the originator check would
return nil.

Andreas has pointed me to the issue he applied it to fix:

http://permalink.gmane.org/gmane.emacs.devel/197339

My plan is to replicate that without the soap-client patch, and try to
fix it a different way, in Debbugs.  From that thread, I think this will
show it:

(async-get (async-start
            `(lambda ()
               (load ,(locate-library "debbugs"))
               (debbugs-get-status 22285))))

I'll work on this hopefully this evening unless you beat me to it.

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> I'll work on this hopefully this evening unless you beat me to it.

Likely, I won't work on this tonight. But we could share the work: you
revert the patch in soap-client.el, and I'll try to adapt debbugs.el
tomorrow over the day. All strings, for which `multibyte-string-p'
returns nil, must be decoded, right?

> Thomas

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Date: Mon, 14 Mar 2016 10:58:44 -0400
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> My plan is to replicate that without the soap-client patch, and try to
> fix it a different way, in Debbugs.  From that thread, I think this will
> show it:
> 
> (async-get (async-start
>             `(lambda ()
>                (load ,(locate-library "debbugs"))
>                (debbugs-get-status 22285))))
> 
> I'll work on this hopefully this evening unless you beat me to it.

Thanks.  But it would be good to discuss what you think should be done
in debbugs.el, before you actually do that.  The important part is to
figure out how can debbugs.el know whether a byte stream is a text
string.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Date: Mon, 14 Mar 2016 10:58:44 -0400
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>> 
>> My plan is to replicate that without the soap-client patch, and try to
>> fix it a different way, in Debbugs.  From that thread, I think this will
>> show it:
>> 
>> (async-get (async-start
>>             `(lambda ()
>>                (load ,(locate-library "debbugs"))
>>                (debbugs-get-status 22285))))
>> 
>> I'll work on this hopefully this evening unless you beat me to it.
>
> Thanks.  But it would be good to discuss what you think should be done
> in debbugs.el, before you actually do that.  The important part is to
> figure out how can debbugs.el know whether a byte stream is a text
> string.

With the soap-client patch reverted, this debbugs.el patch fixes the
above test case, producing a correct multibyte string for originator.

This is just one example, obviously.  The new function should be applied
to all values returned by the Debbugs server that may contain multibyte
UTF-8 characters.  I'll leave this part to Michael.

Also, there's still something not quite right about async, for strings
that don't contain extended characters:

(multibyte-string-p (cdr (assq 'severity (car (async-get (async-start
            `(lambda ()
               (load ,(locate-library "debbugs"))
               (debbugs-get-status 22285))))))))
=> nil

(multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
=> t

but this looks like a side effect of async.

I'll wait to revert the soap-client patch on master and emacs-25 until
Debbugs has a new release to fix this, to avoid temporarily re-breaking
Debbugs.  (However, I'd really like to get this done before Emacs 25.1
so that it doesn't go out with an incompatible soap-client API.)

Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-debbugs-soap-value-to-string.patch --]
[-- Type: text/x-patch, Size: 2038 bytes --]

diff --git a/packages/debbugs/debbugs.el b/packages/debbugs/debbugs.el
index f145280..e4c9667 100644
--- a/packages/debbugs/debbugs.el
+++ b/packages/debbugs/debbugs.el
@@ -264,6 +264,27 @@ (defun debbugs-newest-bugs (amount)
   "Return the list of bug numbers, according to AMOUNT (a number) latest bugs."
   (sort (car (soap-invoke debbugs-wsdl debbugs-port "newest_bugs" amount)) '<))
 
+(defun debbugs-convert-soap-value-to-string (string-value)
+  "If STRING-VALUE is unibyte, decode its contents as a UTF-8 string.
+If STRING-VALUE is a multibyte string, then `soap-client'
+received an xsd:string for this value, and will have decoded it
+already.
+
+If STRING-VALUE is a unibyte string, then `soap-client' received
+an xsd:base64Binary, and ran `base64-decode-string' on it to
+produce a unibyte string of bytes.
+
+For some reason, the Debbugs server code base64-encodes strings
+that contain UTF-8 characters, and returns them as
+xsd:base64Binary, instead of just returning them as xsd:string.
+Therefore, when STRING-VALUE is a unibyte string, we assume its
+bytes represent a UTF-8 string and decode them accordingly."
+  (if (stringp string-value)
+      (if (not (multibyte-string-p string-value))
+	  (decode-coding-string string-value 'utf-8)
+	string-value)
+    (error "Invalid string value")))
+
 (defun debbugs-get-status (&rest bug-numbers)
   "Return a list of status entries for the bugs identified by BUG-NUMBERS.
 
@@ -421,6 +442,11 @@ (defun debbugs-get-status (&rest bug-numbers)
 	    (when (stringp (cdr y))
 	      (setcdr y (mapcar
 			 'string-to-number (split-string (cdr y) " " t)))))
+	  ;; "originator" may be an xsd:base64Binary value containing
+	  ;; a UTF-8-encoded string.
+	  (dolist (attribute '(originator))
+	    (setq y (assoc attribute (cdr (assoc 'value x))))
+	    (setcdr y (debbugs-convert-soap-value-to-string (cdr y))))
 	  ;; "package" is a string, containing comma separated
 	  ;; package names.  "keywords" and "tags" are strings,
 	  ;; containing blank separated package names.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> @@ -421,6 +442,11 @@ (defun debbugs-get-status (&rest bug-numbers)
>  	    (when (stringp (cdr y))
>  	      (setcdr y (mapcar
>  			 'string-to-number (split-string (cdr y) " " t)))))
> +	  ;; "originator" may be an xsd:base64Binary value containing
> +	  ;; a UTF-8-encoded string.

What about all the other strings in the result?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Andreas Schwab <schwab@suse.de> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> @@ -421,6 +442,11 @@ (defun debbugs-get-status (&rest bug-numbers)
>>  	    (when (stringp (cdr y))
>>  	      (setcdr y (mapcar
>>  			 'string-to-number (split-string (cdr y) " " t)))))
>> +	  ;; "originator" may be an xsd:base64Binary value containing
>> +	  ;; a UTF-8-encoded string.
>
> What about all the other strings in the result?

I've added them.

> Andreas.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Also, there's still something not quite right about async, for strings
> that don't contain extended characters:

(multibyte-string-p (read (format "%S" (string-to-multibyte "foo")))) => nil

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> With the soap-client patch reverted, this debbugs.el patch fixes the
> above test case, producing a correct multibyte string for originator.

Thanks. I've committed it in your name.

> This is just one example, obviously.  The new function should be applied
> to all values returned by the Debbugs server that may contain multibyte
> UTF-8 characters.  I'll leave this part to Michael.

Done. The other affected attributes are "subject", "owner" and
"summary"; I've added them.

> Also, there's still something not quite right about async, for strings
> that don't contain extended characters:
>
> (multibyte-string-p (cdr (assq 'severity (car (async-get (async-start
>             `(lambda ()
>                (load ,(locate-library "debbugs"))
>                (debbugs-get-status 22285))))))))
> => nil
>
> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
> => t
>
> but this looks like a side effect of async.

Hmm, don't know. I would like to get rid of async. If
`soap-invoke-async' exists (soap-client >= 3.0), that function is
preferred over `async-start'/`async-get'.

> I'll wait to revert the soap-client patch on master and emacs-25 until
> Debbugs has a new release to fix this, to avoid temporarily re-breaking
> Debbugs.  (However, I'd really like to get this done before Emacs 25.1
> so that it doesn't go out with an incompatible soap-client API.)

I've increased debbugs' version to 0.9.1, shall be released soon by the
elpa release script.

> Thomas

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Michael Albinus <michael.albinus@gmx.de> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> With the soap-client patch reverted, this debbugs.el patch fixes the
>> above test case, producing a correct multibyte string for originator.
>
> Thanks. I've committed it in your name.
>
>> This is just one example, obviously.  The new function should be applied
>> to all values returned by the Debbugs server that may contain multibyte
>> UTF-8 characters.  I'll leave this part to Michael.
>
> Done. The other affected attributes are "subject", "owner" and
> "summary"; I've added them.
>
>> Also, there's still something not quite right about async, for strings
>> that don't contain extended characters:
>>
>> (multibyte-string-p (cdr (assq 'severity (car (async-get (async-start
>>             `(lambda ()
>>                (load ,(locate-library "debbugs"))
>>                (debbugs-get-status 22285))))))))
>> => nil
>>
>> (multibyte-string-p (cdr (assq 'severity (car (debbugs-get-status 22285)))))
>> => t
>>
>> but this looks like a side effect of async.
>
> Hmm, don't know.

I think Andreas explained it with his code snippet.  IIUC, async does a
`read' on the sexp from the child process, and read returns a unibyte
string if there are no non-ASCII characters in the string.  If there are
non-ASCII characters, then read returns a multibyte string:

(let ((pair (read "(\"á\" \"a\")")))
  (list (multibyte-string-p (car pair))
	(multibyte-string-p (cdr pair))))
=> (t nil)

I don't think this will affect debbugs.el's functionality though, since
it will always decode fields that it knows might contain non-ASCII.

The only difference will be that ASCII-only strings coming back from
debbugs-over-async will be unibyte, whereas ASCII-only strings coming
back from in-process debbugs.el will be multibyte, because:

(multibyte-string-p
 (async-get (async-start
	     `(lambda ()
		(decode-coding-string "a" 'utf-8)))))
=> nil

(multibyte-string-p (decode-coding-string "a" 'utf-8))
=> t

As long as no users of the debbugs.el APIs key off multibytedness of the
strings, they'll be fine.  I guess this is a quirk of async, that it
strips the multibytedness of strings that are multibyte-ASCII-only in
the inferior.

> I would like to get rid of async. If `soap-invoke-async' exists
> (soap-client >= 3.0), that function is preferred over
> `async-start'/`async-get'.

The latest soap-client version is also available in GNU ELPA.  Can you
just Package-Requires it, and always rely on the latest version that has
`soap-invoke-async'?

>> I'll wait to revert the soap-client patch on master and emacs-25 until
>> Debbugs has a new release to fix this, to avoid temporarily re-breaking
>> Debbugs.  (However, I'd really like to get this done before Emacs 25.1
>> so that it doesn't go out with an incompatible soap-client API.)
>
> I've increased debbugs' version to 0.9.1, shall be released soon by the
> elpa release script.

Thanks; I should be able to finish the soap-client merge and release
3.1.0 this evening.  Then maybe Debbugs 0.9.2 can Package-Requires
soap-client 3.1.0?

Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> The latest soap-client version is also available in GNU ELPA.  Can you
> just Package-Requires it, and always rely on the latest version that has
> `soap-invoke-async'?

Will do. Last time I've touched debbugs, it was not available in elpa yet.

> Thanks; I should be able to finish the soap-client merge and release
> 3.1.0 this evening.  Then maybe Debbugs 0.9.2 can Package-Requires
> soap-client 3.1.0?

Yep. Tomorrow, when soap-client 3.1.0 is available.

> Thomas

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Michael Albinus <michael.albinus@gmx.de> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> The latest soap-client version is also available in GNU ELPA.  Can you
>> just Package-Requires it, and always rely on the latest version that has
>> `soap-invoke-async'?
>
> Will do. Last time I've touched debbugs, it was not available in elpa yet.
>
>> Thanks; I should be able to finish the soap-client merge and release
>> 3.1.0 this evening.  Then maybe Debbugs 0.9.2 can Package-Requires
>> soap-client 3.1.0?
>
> Yep. Tomorrow, when soap-client 3.1.0 is available.

I synced soap-client 3.1.1 to the Emacs master branch.  But it didn't
get published in the GNU ELPA repository last night.  (I also pushed
Excorporate 0.7.2, which did get published and is now showing as
"incompatible" because of its soap-client 3.1.1 dependency.)

Should updates to :core externals automatically get published to GNU
ELPA?  Does anyone know what I missed or how to debug this?

Thanks,
Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>>
>>> The latest soap-client version is also available in GNU ELPA.  Can you
>>> just Package-Requires it, and always rely on the latest version that has
>>> `soap-invoke-async'?
>>
>> Will do. Last time I've touched debbugs, it was not available in elpa yet.
>>
>>> Thanks; I should be able to finish the soap-client merge and release
>>> 3.1.0 this evening.  Then maybe Debbugs 0.9.2 can Package-Requires
>>> soap-client 3.1.0?
>>
>> Yep. Tomorrow, when soap-client 3.1.0 is available.
>
> I synced soap-client 3.1.1 to the Emacs master branch.  But it didn't
> get published in the GNU ELPA repository last night.  (I also pushed
> Excorporate 0.7.2, which did get published and is now showing as
> "incompatible" because of its soap-client 3.1.1 dependency.)
>
> Should updates to :core externals automatically get published to GNU
> ELPA?  Does anyone know what I missed or how to debug this?

soap-client 3.1.1 is there now.  Not quite sure what I was seeing; maybe
my list-packages test was invalid.  In any case, you should be unblocked
for Debbugs 0.9.2.

Thanks,
Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

Hi Thomas,

> soap-client 3.1.1 is there now.  Not quite sure what I was seeing; maybe
> my list-packages test was invalid.  In any case, you should be unblocked
> for Debbugs 0.9.2.

Thanks. I'll perform some further tests before releasing debbugs
0.9.2. Likely, I'll release it after the Eastern break.

> Thanks,
> Thomas

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> Should updates to :core externals automatically get published to GNU
> ELPA?

Yes (at least those installed in "master").

> Does anyone know what I missed or how to debug this?

I just changed the scripts to give better debug output, so hopefully
next time I'll see what was the problem.  In the mean time, I re-ran the
script and it did make the soap-client package, so the immediate problem
is fixed.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 15 Mar 2016 10:39:02 -0400
> 
> I think Andreas explained it with his code snippet.  IIUC, async does a
> `read' on the sexp from the child process, and read returns a unibyte
> string if there are no non-ASCII characters in the string.  If there are
> non-ASCII characters, then read returns a multibyte string:
> 
> (let ((pair (read "(\"á\" \"a\")")))
>   (list (multibyte-string-p (car pair))
> 	(multibyte-string-p (cdr pair))))
> => (t nil)

Yes.

> I don't think this will affect debbugs.el's functionality though, since
> it will always decode fields that it knows might contain non-ASCII.

Decoding a pure-ASCII string is harmless.

> The only difference will be that ASCII-only strings coming back from
> debbugs-over-async will be unibyte, whereas ASCII-only strings coming
> back from in-process debbugs.el will be multibyte, because:
> 
> (multibyte-string-p
>  (async-get (async-start
> 	     `(lambda ()
> 		(decode-coding-string "a" 'utf-8)))))
> => nil
> 
> (multibyte-string-p (decode-coding-string "a" 'utf-8))
> => t
> 
> As long as no users of the debbugs.el APIs key off multibytedness of the
> strings, they'll be fine.  I guess this is a quirk of async, that it
> strips the multibytedness of strings that are multibyte-ASCII-only in
> the inferior.

I don't think it's something async does, nor is it a quirk.  Depending
on the APIs used, Emacs might decide to produce a unibyte string if
the text is pure ASCII.  That is harmless and should never cause any
problems.

Thanks.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> Date: Mon, 14 Mar 2016 19:58:21 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: Andreas Schwab <schwab@suse.de>, Eli Zaretskii <eliz@gnu.org>, 
> 	Thomas Fitzsimmons <fitzsim@fitzsim.org>, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> The decoding in soap-client is consistent, even though not in the way
> you would like it :-)
> 
>   * if it is told that a value is a string, it will return a string,
>   * if it is told that a value is a byte array it will return a byte
> array (unibyte string)

Told by whom?  By debbugs.el, by the WSDL, by the debbugs server, by
something else?  (Sorry, I know almost nothing about the debbugs or
SOAP.)

> I think the problem here is that the debbugs server encodes utf8
> values as base64, even though the message envelope is utf8 XML and
> could handle them as normal strings.   debbugs.el does not want to
> know about this, so expects strings to be strings.  soap-client.el is
> caught in the middle.

Which ones of the involved parties know, or can know, that the
originator is a string?

> I would also like to reiterate that base64 encoding can be used for
> other things, such as images, and it would not be appropriate to
> decode those as utf8

It should be easy to detect truly binary byte streams: e.g., look for
null bytes.

> (not to mention that such a decoding might fail).

It cannot fail in Emacs, AFAIK.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> The decoding in soap-client is consistent, even though not in the way
>> you would like it :-)
>> 
>>   * if it is told that a value is a string, it will return a string,
>>   * if it is told that a value is a byte array it will return a byte
>> array (unibyte string)
>
> Told by whom?  By debbugs.el, by the WSDL, by the debbugs server, by
> something else?  (Sorry, I know almost nothing about the debbugs or
> SOAP.)

By the XML data returned by the debbugs (SOAP) server.

>> I think the problem here is that the debbugs server encodes utf8
>> values as base64, even though the message envelope is utf8 XML and
>> could handle them as normal strings.   debbugs.el does not want to
>> know about this, so expects strings to be strings.  soap-client.el is
>> caught in the middle.
>
> Which ones of the involved parties know, or can know, that the
> originator is a string?

The debbugs server could know. But is uses the simple-minded SOAP-Lite
Perl library, which encodes everything into base64Binary which doesn't
look like ASCII.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Alex Harsanyi <alexharsanyi@gmail.com>,  schwab@suse.de,  fitzsim@fitzsim.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Mar 2016 19:47:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The decoding in soap-client is consistent, even though not in the way
> >> you would like it :-)
> >> 
> >>   * if it is told that a value is a string, it will return a string,
> >>   * if it is told that a value is a byte array it will return a byte
> >> array (unibyte string)
> >
> > Told by whom?  By debbugs.el, by the WSDL, by the debbugs server, by
> > something else?  (Sorry, I know almost nothing about the debbugs or
> > SOAP.)
> 
> By the XML data returned by the debbugs (SOAP) server.

Then this doesn't really help us, since the server lies to us.

Thanks.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Thomas Fitzsimmons <fitzsim@fitzsim.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Mar 2016 10:15:12 +0100
> 
> As explained in my other message, originator (and other attributes)
> could be sent by the debbugs server as either xsd:string or
> xsd:base64Binary. debbugs.el does not know which encoding has been
> applied, it must trust on consistent soap-client.el decoding.

Does debbugs know the value is a text string?

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> As explained in my other message, originator (and other attributes)
>> could be sent by the debbugs server as either xsd:string or
>> xsd:base64Binary. debbugs.el does not know which encoding has been
>> applied, it must trust on consistent soap-client.el decoding.
>
> Does debbugs know the value is a text string?

debbugs.el knows for every attribute, whether it is a string or
something else. There're not so many attributes, see `debbugs-get-status'.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: schwab@suse.de,  fitzsim@fitzsim.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Mar 2016 19:44:25 +0100
> 
> > Does debbugs know the value is a text string?
> 
> debbugs.el knows for every attribute, whether it is a string or
> something else. There're not so many attributes, see `debbugs-get-status'.

If debbugs.el knows it is a text string, then debbugs.el could decode
it (provided that soap-client.el gives you a unibyte undecoded string
that came out of base64 decoding).

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Does debbugs know the value is a text string?
>> 
>> debbugs.el knows for every attribute, whether it is a string or
>> something else. There're not so many attributes, see `debbugs-get-status'.
>
> If debbugs.el knows it is a text string, then debbugs.el could decode
> it (provided that soap-client.el gives you a unibyte undecoded string
> that came out of base64 decoding).

That's the plan. See the discussion with Thomas.

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Thomas Fitzsimmons]

Andreas Schwab <schwab@suse.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Andreas, is it possible to move the decoding of originator from
>> soap-client to debbugs?
>
> That needs to be answered by its author.

Is there a Debbugs bug (other than the presence of a unibyte string in
debbugs results) that you were trying to fix with your patch to
soap-client?  Like, did you notice the Debbugs user interface
misbehaving, showing undecoded characters to the user, or anything like
that?  Any further information you could provide would be helpful here.

Thanks,
Thomas

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Andreas Schwab]

See this thread: <http://permalink.gmane.org/gmane.emacs.devel/197339>

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

>> Btw, why is "originator" not a string? why xsd:base64Binary?  It's a
>> name of a human (or some other entity), so it's clearly text, no?
>
> Good question, I don't know.  Maybe Michael could comment here, since
> this was a Debbugs decision.

Debbugs.wsdl does not care about the argument types, it regards them as
xsd:anyType. Proper decoding is left to debbugs.el.

On the server side, debbugs.gnu.org, a perl script using SOAP::Lite is
responsible for encoding the attributes. IIUC, the internal package
SOAP::Serializer decides depending on the contents of a string, whether
it shall be xsd:base64Binary, or not. See:

    $self->typelookup({
           'base64Binary' =>
              [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ }, 'as_base64Binary'],
    ...

> Thomas

Best regards, Michael.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Stefan Monnier]

> On the server side, debbugs.gnu.org, a perl script using SOAP::Lite is
> responsible for encoding the attributes. IIUC, the internal package
> SOAP::Serializer decides depending on the contents of a string, whether
> it shall be xsd:base64Binary, or not. See:

>     $self->typelookup({
>            'base64Binary' =>
>               [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ }, 'as_base64Binary'],
>     ...

So on the receiving side, debbugs.el should similarly check
multibyteness of the string to decide what to do with it, I think.


        Stefan

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 14 Mar 2016 08:39:06 -0400
> 
> > On the server side, debbugs.gnu.org, a perl script using SOAP::Lite is
> > responsible for encoding the attributes. IIUC, the internal package
> > SOAP::Serializer decides depending on the contents of a string, whether
> > it shall be xsd:base64Binary, or not. See:
> 
> >     $self->typelookup({
> >            'base64Binary' =>
> >               [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ }, 'as_base64Binary'],
> >     ...
> 
> So on the receiving side, debbugs.el should similarly check
> multibyteness of the string to decide what to do with it, I think.

??? The multibyteness of a string is an illusion created by Emacs, as
you well know.  There's nothing in the byte stream that is inherently
multibyte or unibyte.  The only thing that the multibyteness of the
string could tell us is what kind of processing did soap-client.el do
to produce the string, that's all.

So I'm unsure what you had in mind when you wrote this.  I'm probably
missing something, but what?

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 14 Mar 2016 09:02:48 +0100
> 
> On the server side, debbugs.gnu.org, a perl script using SOAP::Lite is
> responsible for encoding the attributes. IIUC, the internal package
> SOAP::Serializer decides depending on the contents of a string, whether
> it shall be xsd:base64Binary, or not. See:
> 
>     $self->typelookup({
>            'base64Binary' =>
>               [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ }, 'as_base64Binary'],

That doesn't look right: it disallows UTF-8 encoded text in a document
whose encoding is announced to be UTF-8.  But I guess we won't be able
to change that.

Re: emacs-25 b6b47AF: Properly encode/decode base64Binary data in SOAP

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Mon, 14 Mar 2016 09:02:48 +0100
>> 
>> On the server side, debbugs.gnu.org, a perl script using SOAP::Lite is
>> responsible for encoding the attributes. IIUC, the internal package
>> SOAP::Serializer decides depending on the contents of a string, whether
>> it shall be xsd:base64Binary, or not. See:
>> 
>>     $self->typelookup({
>>            'base64Binary' =>
>>               [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ }, 'as_base64Binary'],
>
> That doesn't look right: it disallows UTF-8 encoded text in a document
> whose encoding is announced to be UTF-8.  But I guess we won't be able
> to change that.

No. You would need to write a bug report towards SOAP-Lite. Haven't
check whether this is solved already. Reading existing bug report
titles, there are already some of them wrt utf8. See
<https://rt.cpan.org/Public/Dist/Display.html?Name=SOAP-Lite>.

Best regards, Michael.