From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: pierre.techoueyres@free.fr (Pierre =?utf-8?Q?T=C3=A9choueyres?=) Newsgroups: gmane.emacs.devel Subject: Re: Add support for base64url variant Date: Mon, 27 May 2019 22:30:14 +0200 Message-ID: <87lfyr4p49.fsf@killashandra.ballybran.fr> References: <87pnobiglo.fsf@killashandra.ballybran.fr> <83r28p57jp.fsf@gnu.org> <87zhnd55eq.fsf@killashandra.ballybran.fr> <83mujd54qv.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="116564"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon May 27 22:31:00 2019 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hVMGn-000U5s-EB for ged-emacs-devel@m.gmane.org; Mon, 27 May 2019 22:30:57 +0200 Original-Received: from localhost ([127.0.0.1]:50399 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVMGm-0002Y1-7y for ged-emacs-devel@m.gmane.org; Mon, 27 May 2019 16:30:56 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:55536) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVMGW-0002U5-FL for emacs-devel@gnu.org; Mon, 27 May 2019 16:30:45 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVMGR-0001Mu-8B for emacs-devel@gnu.org; Mon, 27 May 2019 16:30:40 -0400 Original-Received: from smtp4-g21.free.fr ([212.27.42.4]:29094) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVMGI-0001D4-Rc; Mon, 27 May 2019 16:30:27 -0400 Original-Received: from killashandra.ballybran.fr.free.fr (unknown [IPv6:2a01:e0a:1e2:f8b0:9cc3:35bf:aa81:ccf7]) by smtp4-g21.free.fr (Postfix) with ESMTPS id 4D0BA19F59E; Mon, 27 May 2019 22:30:15 +0200 (CEST) In-Reply-To: <83mujd54qv.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 23 May 2019 22:51:20 +0300") X-detected-operating-system: by eggs.gnu.org: Windows NT kernel [generic] [fuzzy] X-Received-From: 212.27.42.4 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:237060 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Eli Zaretskii writes: >> From: pierre.techoueyres@free.fr (Pierre T=C3=A9choueyres) >> Cc: emacs-devel@gnu.org >> Date: Thu, 23 May 2019 21:37:01 +0200 >>=20 >> > I'd suggest to call the new argument base64url or somesuch, since >> > this is trhe official name. >> > >> You mean in replacement of url_variant or b64_value_to_char ? > > The former. > >> How should I send new versions of the patch ? As a full patch in >> attachment like previously ? > > Yes. And please include the log messages (in the ChangeLog style). > > You'll find two patches attached. First one contains requested changes and, hoppefully, a valid Changelog --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Add-support-for-base64url-variant-with-parameters.patch Content-Description: with parameters >From dcd03d67d4d7fa9d1f46554a6703ae562ef39ce5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= Date: Tue, 21 May 2019 23:00:13 +0200 Subject: [PATCH] Add support for base64url variant. Implement the RFC4648 variant of base64 encoding used with url. * doc/lispref/text.texi (base64-encode-region, base64-encode-string, base64-decode-region, base64-decode-string): Adds notice for optionals parameters. * etc/NEWS: Announce support for URL variant of base 64 functions. * src/fns.c (base64-encode-region, base64-encode-string) : Adds optional parameters to manage padding and url variant. (base64-decode-region, base64-decode-string): Adds optional parameter to indicate use of url-variant. (base64-encode-1): Adds parameters to manage padding and url variant. (base64-decode-1): Adds parameter to manage url variant. * test/src/fns-tests.el: Adds tests for encoding / decoding variants. --- doc/lispref/text.texi | 29 ++++++++-- etc/NEWS | 7 +++ src/fns.c | 131 +++++++++++++++++++++++++++++++----------- test/src/fns-tests.el | 131 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 260 insertions(+), 38 deletions(-) diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 278bc3c268..ca1bc4e2df 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4541,10 +4541,10 @@ Base 64 usually written by technical experts acting on their own initiative, and are traditionally written in a pragmatic, experience-driven manner. -}2045. This section describes the functions for +}2045 and also in RFC4648. This section describes the functions for converting to and from this code. -@deffn Command base64-encode-region beg end &optional no-line-break +@deffn Command base64-encode-region beg end &optional no-line-break no-pad base64url This function converts the region from @var{beg} to @var{end} into base 64 code. It returns the length of the encoded text. An error is signaled if a character in the region is multibyte, i.e., in a @@ -4556,9 +4556,15 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the output is just one long line. + +If optional argument @var{no-pad} is set then padding isn't +generated. + +If optional argument @var{base64url} is set, then chars @code{+} and +@code{/} are replaced by @code{-} and @code{_} (See RFC4648). @end deffn -@defun base64-encode-string string &optional no-line-break +@defun base64-encode-string string &optional no-line-break no-pad base64url This function converts the string @var{string} into base 64 code. It returns a string containing the encoded text. As for @code{base64-encode-region}, an error is signaled if a character in the @@ -4568,22 +4574,35 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the result string is just one long line. + +If optional argument @var{no-pad} is non-@code{nil} then padding isn't +generated. + +If optional argument @var{base64url} is non-@code{nil} then chars +@code{+} and @code{/} are replaced by @code{-} abd @code{_} (See +RFC4648). @end defun -@deffn Command base64-decode-region beg end +@deffn Command base64-decode-region beg end &optional base64url This function converts the region from @var{beg} to @var{end} from base 64 code into the corresponding decoded text. It returns the length of the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is non-@code{nil} then padding +become optionnal and url variant is used (See RFC4648). @end deffn -@defun base64-decode-string string +@defun base64-decode-string string &optional base64url This function converts the string @var{string} from base 64 code into the corresponding decoded text. It returns a unibyte string containing the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is non-@code{nil} then padding become +optionnal and url variant is used (See RFC4648). @end defun @node Checksum/Hash diff --git a/etc/NEWS b/etc/NEWS index 222b86ee2b..9d7bfe5e82 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -370,6 +370,13 @@ in tooltips, as it is not useful there. There are 2 new buffer local variables and 1 face to customize this mode they are described in the manual "(emacs) Display". ++++ +** Functions 'base64-(encode|decode)-(string|region)' now manage url variant (RFC4648) +The functions 'base64-encode-(region|string)' now have optionals +arguments to mange padding and url variant. +Mirror function 'base64-decode-(region|string)' now have an optional +argument url-variant to manage this RFC. + * Editing Changes in Emacs 27.1 diff --git a/src/fns.c b/src/fns.c index 6b1f7331f5..963b54fb86 100644 --- a/src/fns.c +++ b/src/fns.c @@ -3169,7 +3169,7 @@ #define MIME_LINE_LENGTH 76 #define IS_ASCII(Character) \ ((Character) < 128) #define IS_BASE64(Character) \ - (IS_ASCII (Character) && base64_char_to_value[Character] >= 0) + (IS_ASCII (Character) && b64_char_to_value[Character] >= 0) #define IS_BASE64_IGNORABLE(Character) \ ((Character) == ' ' || (Character) == '\t' || (Character) == '\n' \ || (Character) == '\f' || (Character) == '\r') @@ -3202,6 +3202,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ '8', '9', '+', '/' /* 60-63 */ }; +static const char base64url_value_to_char[64] = +{ + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', /* 0- 9 */ + 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', /* 10-19 */ + 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', /* 20-29 */ + 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', /* 30-39 */ + 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', /* 40-49 */ + 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', /* 50-59 */ + '8', '9', '-', '_' /* 60-63 */ +}; + /* Table of base64 values for first 128 characters. */ static const short base64_char_to_value[128] = { @@ -3220,6 +3231,23 @@ #define READ_QUADRUPLET_BYTE(retval) \ 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ }; +static const short base64url_char_to_value[128] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 0- 9 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 10- 19 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 20- 29 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 30- 39 */ + -1, -1, -1, -1, -1, 62, -1, -1, 52, 53, /* 40- 49 */ + 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, /* 50- 59 */ + -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, /* 60- 69 */ + 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 70- 79 */ + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, /* 80- 89 */ + 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, /* 90- 99 */ + 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, /* 100-109 */ + 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, /* 110-119 */ + 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ +}; + /* The following diagram shows the logical steps by which three octets get transformed into four base64 characters. @@ -3239,17 +3267,21 @@ #define READ_QUADRUPLET_BYTE(retval) \ base64 characters. */ -static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool); +static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool, + bool, bool); static ptrdiff_t base64_decode_1 (const char *, char *, ptrdiff_t, bool, - ptrdiff_t *); + bool, ptrdiff_t *); DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, - 2, 3, "r", + 2, 5, "r", doc: /* Base64-encode the region between BEG and END. Return the length of the encoded text. Optional third argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break) +into shorter lines. +Optional fourth argument NO-PAD means do not add padding char =. +Optional fifth argument URL-VARIANT means use Url variant (RFC4648). */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object base64url) { char *encoded; ptrdiff_t allength, length; @@ -3273,6 +3305,7 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 ((char *) BYTE_POS_ADDR (ibeg), encoded, length, NILP (no_line_break), + NILP (no_pad), !NILP (base64url), !NILP (BVAR (current_buffer, enable_multibyte_characters))); if (encoded_length > allength) emacs_abort (); @@ -3304,11 +3337,14 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, } DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, - 1, 2, 0, + 1, 4, 0, doc: /* Base64-encode STRING and return the result. Optional second argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object string, Lisp_Object no_line_break) +into shorter lines. +Optional third argument NO-PAD means do not add padding char =. +Optional fourth argument BASE64URL means use Url variant (RFC4648). */) + (Lisp_Object string, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object base64url) { ptrdiff_t allength, length, encoded_length; char *encoded; @@ -3329,6 +3365,7 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, encoded_length = base64_encode_1 (SSDATA (string), encoded, length, NILP (no_line_break), + NILP (no_pad), !NILP (base64url), STRING_MULTIBYTE (string)); if (encoded_length > allength) emacs_abort (); @@ -3347,7 +3384,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, static ptrdiff_t base64_encode_1 (const char *from, char *to, ptrdiff_t length, - bool line_break, bool multibyte) + bool line_break, bool pad, bool base64url, + bool multibyte) { int counter = 0; ptrdiff_t i = 0; @@ -3355,6 +3393,7 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, int c; unsigned int value; int bytes; + char const *b64_value_to_char = (base64url) ? base64url_value_to_char : base64_value_to_char; while (i < length) { @@ -3385,16 +3424,19 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, /* Process first byte of a triplet. */ - *e++ = base64_value_to_char[0x3f & c >> 2]; + *e++ = b64_value_to_char[0x3f & c >> 2]; value = (0x03 & c) << 4; /* Process second byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + *e++ = '='; + } break; } @@ -3410,15 +3452,18 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x0f & c >> 4)]; + *e++ = b64_value_to_char[value | (0x0f & c >> 4)]; value = (0x0f & c) << 2; /* Process third byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + } break; } @@ -3434,8 +3479,8 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x03 & c >> 6)]; - *e++ = base64_value_to_char[0x3f & c]; + *e++ = b64_value_to_char[value | (0x03 & c >> 6)]; + *e++ = b64_value_to_char[0x3f & c]; } return e - to; @@ -3443,11 +3488,13 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, - 2, 2, "r", + 2, 3, "r", doc: /* Base64-decode the region between BEG and END. Return the length of the decoded text. -If the region can't be decoded, signal an error and don't modify the buffer. */) - (Lisp_Object beg, Lisp_Object end) +If the region can't be decoded, signal an error and don't modify the buffer. +Optional third argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object base64url) { ptrdiff_t ibeg, iend, length, allength; char *decoded; @@ -3472,7 +3519,7 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, move_gap_both (XFIXNAT (beg), ibeg); decoded_length = base64_decode_1 ((char *) BYTE_POS_ADDR (ibeg), - decoded, length, + decoded, length, !NILP (base64url), multibyte, &inserted_chars); if (decoded_length > allength) emacs_abort (); @@ -3506,9 +3553,11 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, } DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, - 1, 1, 0, - doc: /* Base64-decode STRING and return the result. */) - (Lisp_Object string) + 1, 2, 0, + doc: /* Base64-decode STRING and return the result +Optional argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object string, Lisp_Object base64url) { char *decoded; ptrdiff_t length, decoded_length; @@ -3523,7 +3572,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, /* The decoded result should be unibyte. */ decoded_length = base64_decode_1 (SSDATA (string), decoded, length, - 0, NULL); + !NILP (base64url), 0, NULL); if (decoded_length > length) emacs_abort (); else if (decoded_length >= 0) @@ -3545,6 +3594,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, static ptrdiff_t base64_decode_1 (const char *from, char *to, ptrdiff_t length, + bool base64url, bool multibyte, ptrdiff_t *nchars_return) { ptrdiff_t i = 0; /* Used inside READ_QUADRUPLET_BYTE */ @@ -3552,6 +3602,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, unsigned char c; unsigned long value; ptrdiff_t nchars = 0; + short const *b64_char_to_value = (base64url) ? base64url_char_to_value : base64_char_to_value; while (1) { @@ -3561,7 +3612,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value = base64_char_to_value[c] << 18; + value = b64_char_to_value[c] << 18; /* Process second byte of a quadruplet. */ @@ -3569,7 +3620,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 12; + value |= b64_char_to_value[c] << 12; c = (unsigned char) (value >> 16); if (multibyte && c >= 128) @@ -3580,7 +3631,14 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process third byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') { @@ -3593,7 +3651,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 6; + value |= b64_char_to_value[c] << 6; c = (unsigned char) (0xff & value >> 8); if (multibyte && c >= 128) @@ -3604,14 +3662,21 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process fourth byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') continue; if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c]; + value |= b64_char_to_value[c]; c = (unsigned char) (0xff & value); if (multibyte && c >= 128) diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el index 6ebab4287f..3301129567 100644 --- a/test/src/fns-tests.el +++ b/test/src/fns-tests.el @@ -233,6 +233,137 @@ fns-tests-func-arity (should (equal (func-arity (eval (lambda (x &optional y)) t)) '(1 . 2))) (should (equal (func-arity 'let) '(1 . unevalled)))) +(defun string-repeat (s o) + (apply 'concat (make-list o s))) + +(ert-deftest fns-tests-base64-encode-string () + ;; standard variant RFC2045 + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f") "Zg==")) + (should (equal (base64-encode-string "fo") "Zm8=")) + (should (equal (base64-encode-string "foo") "Zm9v")) + (should (equal (base64-encode-string "foob") "Zm9vYg==")) + (should (equal (base64-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/")) + + ;; no line break + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string (string-repeat "f" 100) t) (concat (string-repeat "Zm" 66) "Zg=="))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw=="))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) (string-repeat "FPucA9l+" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) (string-repeat "FPucA9l/" 10))) + + ;; no paddign + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f" nil t) "Zg")) + (should (equal (base64-encode-string "fo" nil t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t) "Zm9vYmFy")) + + ;; url variant wih padding + (should (equal (base64-encode-string "" nil nil t) "")) + (should (equal (base64-encode-string "f" nil nil t) "Zg==")) + (should (equal (base64-encode-string "fo" nil nil t) "Zm8=")) + (should (equal (base64-encode-string "foo" nil nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil nil t) "Zm9vYg==")) + (should (equal (base64-encode-string "fooba" nil nil t) "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar" nil nil t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil nil t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil nil t) "FPucA9l_")) + + ;; url variant no padding + (should (equal (base64-encode-string "" nil t t) "")) + (should (equal (base64-encode-string "f" nil t t) "Zg")) + (should (equal (base64-encode-string "fo" nil t t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil t t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil t t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (base64-encode-string (string-repeat "f" 100) t t t) (concat (string-repeat "Zm" 66) "Zg"))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t t t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t t t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t t t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t t t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t t t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t t t) (string-repeat "FPucA9l-" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t t t) (string-repeat "FPucA9l_" 10))) + + ) + +(ert-deftest fns-tests-base64-decode-string () + ;; standard variant RFC2045 + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==") "f")) + (should (equal (base64-decode-string "Zm8=") "fo")) + (should (equal (base64-decode-string "Zm9v") "foo")) + (should (equal (base64-decode-string "Zm9vYg==") "foob")) + (should (equal (base64-decode-string "Zm9vYmE=") "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy") "foobar")) + (should (equal (base64-decode-string "FPucA9l+") "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l/") "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; no paddign + (should (equal (base64-decode-string "" t) "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + + ;; url variant wih padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==" t) "f") ) + (should (equal (base64-decode-string "Zm8=" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg==" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE=" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; url variant no padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + + ;; url variant no line break no padding + (should (equal (base64-decode-string (concat (string-repeat "Zm" 66) "Zg") t) (string-repeat "f" 100))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw") t) (string-repeat "fo" 50))) + (should (equal (base64-decode-string (string-repeat "Zm9v" 25) t) (string-repeat "foo" 25))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmZvb2Jmb29i" 5) t) (string-repeat "foob" 15))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5) t) (string-repeat "fooba" 15))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy") t) (string-repeat "foobar" 15))) + (should (equal (base64-decode-string (string-repeat "FPucA9l-" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10))) + (should (equal (base64-decode-string (string-repeat "FPucA9l_" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10))) + + ;; errors check + (should (eq :got-error (condition-case () (base64-decode-string "Zg=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmE") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmFy=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zg=Zg=") (error :got-error)))) + ) + (ert-deftest fns-tests-hash-buffer () (should (equal (sha1 "foo") "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33")) (should (equal (with-temp-buffer -- 2.21.0 --=-=-= Content-Type: text/plain The second patch contains an reworked version wich instead of adding many parameters to base64-encode-region (resp. base64-encode-string) function create the base64url-encode-region (resp. base64url-encode-string) function. Documentation and tests are also updated consequently. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Add-support-for-base64url-variant-with-new-functions.patch Content-Description: new functions >From 8acbb905aeb49fb0bfb747acc6b4965cf120e23b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= Date: Tue, 21 May 2019 23:00:13 +0200 Subject: [PATCH] Add support for base64url variant. Implements the RFC4648 variant of base64 encoding used by URLs. * doc/lispref/text.texi (base64url-encode-region, base64url-encode-string): Documents new functions. (base64-decode-region, base64-decode-string): Documents new optional parameter 'base64url' used to use url variant when decoding data. * etc/NEWS: Announce new functions and optionals parameters. * src/fns.c (base64url-encode-region, base64url-encode-region): New functions to manage url variant. (base64-decode-region, base64-decode-string): Adds optional parameter to indicate use of url-variant. (base64_encode_region_1, base64_encode_string_1): Internal functions with extracted code from 'base64_encode_region' and 'base64_encode_string' and optional parameters to manage padding and url variant. (base64-encode-region, base64-encode-string) : Use internal functions previously defined. (base64-encode-1): Adds parameters to manage padding and url variant. (base64-decode-1): Adds parameter to manage url variant. * test/src/fns-tests.el (fns-tests--with-region): New helper macro to test region variant of base64 encode / decode functions. (fns-tests--string-repeat): Helper function used in base64 tests. (fns-tests-base64-encode-region, fns-tests-base64-encode-string): Tests for standard base64 function. (fns-test-base64url-encode-region, fns-test-base64url-encode-string): Tests for url variant. (fns-tests-base64-decode-string): Tests for decoding part. --- doc/lispref/text.texi | 42 +++++++++- etc/NEWS | 8 ++ src/fns.c | 166 ++++++++++++++++++++++++++++++++------- test/src/fns-tests.el | 179 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 363 insertions(+), 32 deletions(-) diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 278bc3c268..3e0cf4c06f 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4541,7 +4541,7 @@ Base 64 usually written by technical experts acting on their own initiative, and are traditionally written in a pragmatic, experience-driven manner. -}2045. This section describes the functions for +}2045 and also in RFC4648. This section describes the functions for converting to and from this code. @deffn Command base64-encode-region beg end &optional no-line-break @@ -4558,6 +4558,22 @@ Base 64 the output is just one long line. @end deffn +@deffn Command base64url-encode-region beg end &optional no-pad +This function converts the region from @var{beg} to @var{end} into base +64 code. It returns the length of the encoded text. An error is +signaled if a character in the region is multibyte, i.e., in a +multibyte buffer the region must contain only characters from the +charsets @code{ascii}, @code{eight-bit-control} and +@code{eight-bit-graphic}. + +Contrary to the function @code{base64-encode-region}, this function +doesnt inserts newline characters into the encoded text, so the output +is just one long line. + +If the optional argument @var{no-pad} is non-@code{nil} then padding +(@code{=}) isn't generated. +@end deffn + @defun base64-encode-string string &optional no-line-break This function converts the string @var{string} into base 64 code. It returns a string containing the encoded text. As for @@ -4570,20 +4586,40 @@ Base 64 the result string is just one long line. @end defun -@deffn Command base64-decode-region beg end +@defun base64url-encode-string string &optional no-pad +This function converts the string @var{string} into base 64 url code +(see RFC4648). It returns a string containing the encoded text. As +for @code{base64url-encode-region}, an error is signaled if a +character in the string is multibyte. + +Contrary to @code{base64-encode-string}, this function doesnt inserts +newline characters into the encoded text, so the result string is just +one long line. + +If the optional argument @var{no-pad} is non-@code{nil} then padding +(@code{=}) isn't generated. +@end defun + +@deffn Command base64-decode-region beg end &optional base64url This function converts the region from @var{beg} to @var{end} from base 64 code into the corresponding decoded text. It returns the length of the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is is non-@code{nil} then padding +become optionnal and url variant is used (see RFC4648). @end deffn -@defun base64-decode-string string +@defun base64-decode-string string &optional base64url This function converts the string @var{string} from base 64 code into the corresponding decoded text. It returns a unibyte string containing the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is is non-@code{nil} then padding +become optionnal and url variant is used (see RFC4648). @end defun @node Checksum/Hash diff --git a/etc/NEWS b/etc/NEWS index 222b86ee2b..cae6b25aca 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -370,6 +370,14 @@ in tooltips, as it is not useful there. There are 2 new buffer local variables and 1 face to customize this mode they are described in the manual "(emacs) Display". ++++ +** New functions 'base64url-encode-(string|region)' manage url variant (RFC4648) +The new functions 'base64url-encode-region' and +'base64url-encode-region' now implements url-variant as defined in RFC. + +Mirror functions 'base64-decode-(region|string)' now have an optional +argument base64url to manage this RFC. + * Editing Changes in Emacs 27.1 diff --git a/src/fns.c b/src/fns.c index 6b1f7331f5..8c6894b77d 100644 --- a/src/fns.c +++ b/src/fns.c @@ -3169,7 +3169,7 @@ #define MIME_LINE_LENGTH 76 #define IS_ASCII(Character) \ ((Character) < 128) #define IS_BASE64(Character) \ - (IS_ASCII (Character) && base64_char_to_value[Character] >= 0) + (IS_ASCII (Character) && b64_char_to_value[Character] >= 0) #define IS_BASE64_IGNORABLE(Character) \ ((Character) == ' ' || (Character) == '\t' || (Character) == '\n' \ || (Character) == '\f' || (Character) == '\r') @@ -3202,6 +3202,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ '8', '9', '+', '/' /* 60-63 */ }; +static const char base64url_value_to_char[64] = +{ + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', /* 0- 9 */ + 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', /* 10-19 */ + 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', /* 20-29 */ + 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', /* 30-39 */ + 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', /* 40-49 */ + 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', /* 50-59 */ + '8', '9', '-', '_' /* 60-63 */ +}; + /* Table of base64 values for first 128 characters. */ static const short base64_char_to_value[128] = { @@ -3220,6 +3231,23 @@ #define READ_QUADRUPLET_BYTE(retval) \ 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ }; +static const short base64url_char_to_value[128] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 0- 9 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 10- 19 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 20- 29 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 30- 39 */ + -1, -1, -1, -1, -1, 62, -1, -1, 52, 53, /* 40- 49 */ + 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, /* 50- 59 */ + -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, /* 60- 69 */ + 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 70- 79 */ + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, /* 80- 89 */ + 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, /* 90- 99 */ + 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, /* 100-109 */ + 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, /* 110-119 */ + 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ +}; + /* The following diagram shows the logical steps by which three octets get transformed into four base64 characters. @@ -3239,9 +3267,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ base64 characters. */ -static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool); +static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool, + bool, bool); static ptrdiff_t base64_decode_1 (const char *, char *, ptrdiff_t, bool, - ptrdiff_t *); + bool, ptrdiff_t *); + +Lisp_Object base64_encode_region_1 (Lisp_Object, Lisp_Object, bool, + bool, bool); + +Lisp_Object base64_encode_string_1(Lisp_Object, bool, + bool, bool); + DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, 2, 3, "r", @@ -3250,6 +3286,26 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, Optional third argument NO-LINE-BREAK means do not break long lines into shorter lines. */) (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break) +{ + return base64_encode_region_1(beg, end, NILP (no_line_break), true, false); +} + + +DEFUN ("base64url-encode-region", Fbase64url_encode_region, Sbase64url_encode_region, + 2, 3, "r", + doc: /* Base64url-encode the region between BEG and END. +Return the length of the encoded text. +Optional second argument NO-PAD means do not add padding char =. + +This is the variant defined in RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object no_pad) +{ + return base64_encode_region_1(beg, end, false, NILP(no_pad), true); +} + +Lisp_Object +base64_encode_region_1 (Lisp_Object beg, Lisp_Object end, bool line_break, + bool pad, bool base64url) { char *encoded; ptrdiff_t allength, length; @@ -3272,7 +3328,8 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 ((char *) BYTE_POS_ADDR (ibeg), - encoded, length, NILP (no_line_break), + encoded, length, line_break, + pad, base64url, !NILP (BVAR (current_buffer, enable_multibyte_characters))); if (encoded_length > allength) emacs_abort (); @@ -3310,6 +3367,26 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, into shorter lines. */) (Lisp_Object string, Lisp_Object no_line_break) { + + return base64_encode_string_1(string, NILP (no_line_break), true, false); +} + +DEFUN ("base64url-encode-string", Fbase64url_encode_string, Sbase64url_encode_string, + 1, 2, 0, + doc: /* Base64url-encode STRING and return the result. +Optional second argument NO-PAD means do not add padding char =. + +This is the variant defined in RFC4648. */) + (Lisp_Object string, Lisp_Object no_pad) +{ + + return base64_encode_string_1(string, false, NILP(no_pad), true); +} + +Lisp_Object +base64_encode_string_1(Lisp_Object string, bool line_break, + bool pad, bool base64url) +{ ptrdiff_t allength, length, encoded_length; char *encoded; Lisp_Object encoded_string; @@ -3328,7 +3405,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 (SSDATA (string), - encoded, length, NILP (no_line_break), + encoded, length, line_break, + pad, base64url, STRING_MULTIBYTE (string)); if (encoded_length > allength) emacs_abort (); @@ -3347,7 +3425,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, static ptrdiff_t base64_encode_1 (const char *from, char *to, ptrdiff_t length, - bool line_break, bool multibyte) + bool line_break, bool pad, bool base64url, + bool multibyte) { int counter = 0; ptrdiff_t i = 0; @@ -3355,6 +3434,7 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, int c; unsigned int value; int bytes; + char const *b64_value_to_char = (base64url) ? base64url_value_to_char : base64_value_to_char; while (i < length) { @@ -3385,16 +3465,19 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, /* Process first byte of a triplet. */ - *e++ = base64_value_to_char[0x3f & c >> 2]; + *e++ = b64_value_to_char[0x3f & c >> 2]; value = (0x03 & c) << 4; /* Process second byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + *e++ = '='; + } break; } @@ -3410,15 +3493,18 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x0f & c >> 4)]; + *e++ = b64_value_to_char[value | (0x0f & c >> 4)]; value = (0x0f & c) << 2; /* Process third byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + } break; } @@ -3434,8 +3520,8 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x03 & c >> 6)]; - *e++ = base64_value_to_char[0x3f & c]; + *e++ = b64_value_to_char[value | (0x03 & c >> 6)]; + *e++ = b64_value_to_char[0x3f & c]; } return e - to; @@ -3443,11 +3529,13 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, - 2, 2, "r", + 2, 3, "r", doc: /* Base64-decode the region between BEG and END. Return the length of the decoded text. -If the region can't be decoded, signal an error and don't modify the buffer. */) - (Lisp_Object beg, Lisp_Object end) +If the region can't be decoded, signal an error and don't modify the buffer. +Optional third argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object base64url) { ptrdiff_t ibeg, iend, length, allength; char *decoded; @@ -3472,7 +3560,7 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, move_gap_both (XFIXNAT (beg), ibeg); decoded_length = base64_decode_1 ((char *) BYTE_POS_ADDR (ibeg), - decoded, length, + decoded, length, !NILP (base64url), multibyte, &inserted_chars); if (decoded_length > allength) emacs_abort (); @@ -3506,9 +3594,11 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, } DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, - 1, 1, 0, - doc: /* Base64-decode STRING and return the result. */) - (Lisp_Object string) + 1, 2, 0, + doc: /* Base64-decode STRING and return the result +Optional argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object string, Lisp_Object base64url) { char *decoded; ptrdiff_t length, decoded_length; @@ -3523,7 +3613,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, /* The decoded result should be unibyte. */ decoded_length = base64_decode_1 (SSDATA (string), decoded, length, - 0, NULL); + !NILP (base64url), 0, NULL); if (decoded_length > length) emacs_abort (); else if (decoded_length >= 0) @@ -3545,6 +3635,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, static ptrdiff_t base64_decode_1 (const char *from, char *to, ptrdiff_t length, + bool base64url, bool multibyte, ptrdiff_t *nchars_return) { ptrdiff_t i = 0; /* Used inside READ_QUADRUPLET_BYTE */ @@ -3552,6 +3643,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, unsigned char c; unsigned long value; ptrdiff_t nchars = 0; + short const *b64_char_to_value = (base64url) ? base64url_char_to_value : base64_char_to_value; while (1) { @@ -3561,7 +3653,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value = base64_char_to_value[c] << 18; + value = b64_char_to_value[c] << 18; /* Process second byte of a quadruplet. */ @@ -3569,7 +3661,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 12; + value |= b64_char_to_value[c] << 12; c = (unsigned char) (value >> 16); if (multibyte && c >= 128) @@ -3580,7 +3672,14 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process third byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') { @@ -3593,7 +3692,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 6; + value |= b64_char_to_value[c] << 6; c = (unsigned char) (0xff & value >> 8); if (multibyte && c >= 128) @@ -3604,14 +3703,21 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process fourth byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') continue; if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c]; + value |= b64_char_to_value[c]; c = (unsigned char) (0xff & value); if (multibyte && c >= 128) @@ -5445,6 +5551,8 @@ syms_of_fns (void) defsubr (&Sbase64_decode_region); defsubr (&Sbase64_encode_string); defsubr (&Sbase64_decode_string); + defsubr (&Sbase64url_encode_region); + defsubr (&Sbase64url_encode_string); defsubr (&Smd5); defsubr (&Ssecure_hash_algorithms); defsubr (&Ssecure_hash); diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el index 6ebab4287f..dab43212f5 100644 --- a/test/src/fns-tests.el +++ b/test/src/fns-tests.el @@ -233,6 +233,185 @@ fns-tests-func-arity (should (equal (func-arity (eval (lambda (x &optional y)) t)) '(1 . 2))) (should (equal (func-arity 'let) '(1 . unevalled)))) +(defun fns-tests--string-repeat (s o) + (apply 'concat (make-list o s))) + +(defmacro fns-tests--with-region (funcname string &rest args) + "Apply FUNCNAME in a temp bufer on the region produced by STRING." + (declare (indent 1)) + `(with-temp-buffer + (insert ,string) + (,funcname (point-min) (point-max) ,@args) + (buffer-string))) + +(ert-deftest fns-tests-base64-encode-region () + ;; standard variant RFC2045 + (should (equal (fns-tests--with-region base64-encode-region "") "")) + (should (equal (fns-tests--with-region base64-encode-region "f") "Zg==")) + (should (equal (fns-tests--with-region base64-encode-region "fo") "Zm8=")) + (should (equal (fns-tests--with-region base64-encode-region "foo") "Zm9v")) + (should (equal (fns-tests--with-region base64-encode-region "foob") "Zm9vYg==")) + (should (equal (fns-tests--with-region base64-encode-region "fooba") "Zm9vYmE=")) + (should (equal (fns-tests--with-region base64-encode-region "foobar") "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64-encode-region "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (fns-tests--with-region base64-encode-region "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/"))) + +(ert-deftest fns-tests-base64-encode-string () + ;; standard variant RFC2045 + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f") "Zg==")) + (should (equal (base64-encode-string "fo") "Zm8=")) + (should (equal (base64-encode-string "foo") "Zm9v")) + (should (equal (base64-encode-string "foob") "Zm9vYg==")) + (should (equal (base64-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/"))) + +(ert-deftest fns-test-base64url-encode-region () + ;; url variant wih padding + (should (equal (fns-tests--with-region base64url-encode-region "") "")) + (should (equal (fns-tests--with-region base64url-encode-region "f") "Zg==")) + (should (equal (fns-tests--with-region base64url-encode-region "fo") "Zm8=")) + (should (equal (fns-tests--with-region base64url-encode-region "foo") "Zm9v")) + (should (equal (fns-tests--with-region base64url-encode-region "foob") "Zm9vYg==")) + (should (equal (fns-tests--with-region base64url-encode-region "fooba") "Zm9vYmE=")) + (should (equal (fns-tests--with-region base64url-encode-region "foobar") "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l-")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l_")) + + ;; url variant no padding + (should (equal (fns-tests--with-region base64url-encode-region "" t) "")) + (should (equal (fns-tests--with-region base64url-encode-region "f" t) "Zg")) + (should (equal (fns-tests--with-region base64url-encode-region "fo" t) "Zm8")) + (should (equal (fns-tests--with-region base64url-encode-region "foo" t) "Zm9v")) + (should (equal (fns-tests--with-region base64url-encode-region "foob" t) "Zm9vYg")) + (should (equal (fns-tests--with-region base64url-encode-region "fooba" t) "Zm9vYmE")) + (should (equal (fns-tests--with-region base64url-encode-region "foobar" t) "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7e" t) "FPucA9l-")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7f" t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "f" 100) t) + (concat (fns-tests--string-repeat "Zm" 66) "Zg"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "fo" 50) t) + (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foo" 25) t) + (fns-tests--string-repeat "Zm9v" 25))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foob" 15) t) + (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "fooba" 15) t) + (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foobar" 15) t) + (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) + (fns-tests--string-repeat "FPucA9l-" 10))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) + (fns-tests--string-repeat "FPucA9l_" 10)))) + +(ert-deftest fns-test-base64url-encode-string () + ;; url variant wih padding + (should (equal (base64url-encode-string "") "")) + (should (equal (base64url-encode-string "f") "Zg==")) + (should (equal (base64url-encode-string "fo") "Zm8=")) + (should (equal (base64url-encode-string "foo") "Zm9v")) + (should (equal (base64url-encode-string "foob") "Zm9vYg==")) + (should (equal (base64url-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64url-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l-")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l_")) + + ;; url variant no padding + (should (equal (base64url-encode-string "" t) "")) + (should (equal (base64url-encode-string "f" t) "Zg")) + (should (equal (base64url-encode-string "fo" t) "Zm8")) + (should (equal (base64url-encode-string "foo" t) "Zm9v")) + (should (equal (base64url-encode-string "foob" t) "Zm9vYg")) + (should (equal (base64url-encode-string "fooba" t) "Zm9vYmE")) + (should (equal (base64url-encode-string "foobar" t) "Zm9vYmFy")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7e" t) "FPucA9l-")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7f" t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (base64url-encode-string (fns-tests--string-repeat "f" 100) t) (concat (fns-tests--string-repeat "Zm" 66) "Zg"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "fo" 50) t) (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foo" 25) t) (fns-tests--string-repeat "Zm9v" 25))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foob" 15) t) (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "fooba" 15) t) (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foobar" 15) t) (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) (fns-tests--string-repeat "FPucA9l-" 10))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) (fns-tests--string-repeat "FPucA9l_" 10)))) + +(ert-deftest fns-tests-base64-decode-string () + ;; standard variant RFC2045 + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==") "f")) + (should (equal (base64-decode-string "Zm8=") "fo")) + (should (equal (base64-decode-string "Zm9v") "foo")) + (should (equal (base64-decode-string "Zm9vYg==") "foob")) + (should (equal (base64-decode-string "Zm9vYmE=") "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy") "foobar")) + (should (equal (base64-decode-string "FPucA9l+") "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l/") "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; no paddign + (should (equal (base64-decode-string "" t) "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + + ;; url variant wih padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==" t) "f") ) + (should (equal (base64-decode-string "Zm8=" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg==" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE=" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; url variant no padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + + ;; url variant no line break no padding + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm" 66) "Zg") t) + (fns-tests--string-repeat "f" 100))) + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw") t) + (fns-tests--string-repeat "fo" 50))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9v" 25) t) + (fns-tests--string-repeat "foo" 25))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5) t) + (fns-tests--string-repeat "foob" 15))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5) t) + (fns-tests--string-repeat "fooba" 15))) + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy") t) + (fns-tests--string-repeat "foobar" 15))) + (should (equal (base64-decode-string (fns-tests--string-repeat "FPucA9l-" 10) t) + (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10))) + (should (equal (base64-decode-string (fns-tests--string-repeat "FPucA9l_" 10) t) + (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10))) + + ;; errors check + (should (eq :got-error (condition-case () (base64-decode-string "Zg=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmE") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmFy=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zg=Zg=") (error :got-error))))) + (ert-deftest fns-tests-hash-buffer () (should (equal (sha1 "foo") "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33")) (should (equal (with-temp-buffer -- 2.21.0 --=-=-= Content-Type: text/plain Tell me what you thing is the way to go. --=-=-=--