* Add support for base64url variant @ 2019-05-21 22:32 Pierre Téchoueyres 2019-05-22 7:43 ` Eli Zaretskii 2019-05-23 18:50 ` Eli Zaretskii 0 siblings, 2 replies; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-21 22:32 UTC (permalink / raw) To: Emacs developers [-- Attachment #1: Type: text/plain, Size: 439 bytes --] I'm actually working on an emacs library to work with JWT (JSON Web Token) as described in RFC 7519 / 7797. To achive that I must use an variant of base 64 encoding named base64url as described in RFC4648. Actually Emacs doesn't provide this variant. I've attached an attempt to implement it at C level. I'm avare that my patch is far from perfect and I would welcome any help or comments to improve it. Thanks in advance. Pierre. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: base64url --] [-- Type: text/x-patch, Size: 24812 bytes --] From 55a38a1d32758ac31746c2c9c8d54934f36c55ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr> Date: Tue, 21 May 2019 23:00:13 +0200 Subject: [PATCH] Add support for base64url variant. Implement the RFC4648 variant of base64 encoding used in URL (base64url). * doc/lispref/text.texi: add notice for optionals parameters of base64-(encode|decode)-(string|region). * etc/NEWS: announce variants for previous functions. * src/fns.c (base64-encode-region, base64-encode-string) : add optional parameters to manage padding and url variant. (base64-decode-region, base64-decode-string): add optional parameter to indicate use of url-variant. (base64-encode-1): add parameters to manage padding and url variant. (base64-decode-1): add parameter to manage url variant. * test/src/fns-tests.el: add tests for encoding / decoding variants. --- doc/lispref/text.texi | 28 +++++++-- etc/NEWS | 7 +++ src/fns.c | 129 ++++++++++++++++++++++++++++++----------- test/src/fns-tests.el | 131 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 257 insertions(+), 38 deletions(-) diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 278bc3c268..30b458c19e 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4541,10 +4541,10 @@ Base 64 usually written by technical experts acting on their own initiative, and are traditionally written in a pragmatic, experience-driven manner. -}2045. This section describes the functions for +}2045 and also in RFC4648. This section describes the functions for converting to and from this code. -@deffn Command base64-encode-region beg end &optional no-line-break +@deffn Command base64-encode-region beg end &optional no-line-break no-pad url-variant This function converts the region from @var{beg} to @var{end} into base 64 code. It returns the length of the encoded text. An error is signaled if a character in the region is multibyte, i.e., in a @@ -4556,9 +4556,15 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the output is just one long line. + +If optional argument @var{no-pad} is set then padding isn't +generated. + +If optional argument @var{url-variant} is set, then chars @code{+} and +@code{/} are replaced by @code{-} and @code{_} (See RFC4648). @end deffn -@defun base64-encode-string string &optional no-line-break +@defun base64-encode-string string &optional no-line-break no-pad url-variant This function converts the string @var{string} into base 64 code. It returns a string containing the encoded text. As for @code{base64-encode-region}, an error is signaled if a character in the @@ -4568,22 +4574,34 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the result string is just one long line. + +If optional argument @var{no-pad} is set then padding isn't +generated. + +If optional argument @var{url-variant} is set then chars @code{+} and +@code{/} are replaced by @code{-} abd @code{_} (See RFC4648). @end defun -@deffn Command base64-decode-region beg end +@deffn Command base64-decode-region beg end &optional url-variant This function converts the region from @var{beg} to @var{end} from base 64 code into the corresponding decoded text. It returns the length of the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{url-variant} is set then padding become +optionnal and url variant is used (See RFC4648). @end deffn -@defun base64-decode-string string +@defun base64-decode-string string &optional url-variant This function converts the string @var{string} from base 64 code into the corresponding decoded text. It returns a unibyte string containing the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{url-variant} is set then padding become +optionnal and url variant is used (See RFC4648). @end defun @node Checksum/Hash diff --git a/etc/NEWS b/etc/NEWS index 72702a9aaa..a87c7d140a 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -364,6 +364,13 @@ in tooltips, as it is not useful there. There are 2 new buffer local variables and 1 face to customize this mode they are described in the manual "(emacs) Display". ++++ +** Functions 'base64-(encode|decode)-(string|region)' now manage url variant (RFC4648) +The functions 'base64-encode-(region|string)' now have optionals +arguments to mange padding and url-variant. +Mirror function 'base64-decode-(region|string)' now have an optional +argument url-variant to manage this RFC. + \f * Editing Changes in Emacs 27.1 diff --git a/src/fns.c b/src/fns.c index c3202495da..f868f13214 100644 --- a/src/fns.c +++ b/src/fns.c @@ -3171,7 +3171,7 @@ #define MIME_LINE_LENGTH 76 #define IS_ASCII(Character) \ ((Character) < 128) #define IS_BASE64(Character) \ - (IS_ASCII (Character) && base64_char_to_value[Character] >= 0) + (IS_ASCII (Character) && b64_char_to_value[Character] >= 0) #define IS_BASE64_IGNORABLE(Character) \ ((Character) == ' ' || (Character) == '\t' || (Character) == '\n' \ || (Character) == '\f' || (Character) == '\r') @@ -3204,6 +3204,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ '8', '9', '+', '/' /* 60-63 */ }; +static const char base64url_value_to_char[64] = +{ + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', /* 0- 9 */ + 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', /* 10-19 */ + 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', /* 20-29 */ + 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', /* 30-39 */ + 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', /* 40-49 */ + 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', /* 50-59 */ + '8', '9', '-', '_' /* 60-63 */ +}; + /* Table of base64 values for first 128 characters. */ static const short base64_char_to_value[128] = { @@ -3222,6 +3233,23 @@ #define READ_QUADRUPLET_BYTE(retval) \ 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ }; +static const short base64url_char_to_value[128] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 0- 9 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 10- 19 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 20- 29 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 30- 39 */ + -1, -1, -1, -1, -1, 62, -1, -1, 52, 53, /* 40- 49 */ + 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, /* 50- 59 */ + -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, /* 60- 69 */ + 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 70- 79 */ + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, /* 80- 89 */ + 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, /* 90- 99 */ + 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, /* 100-109 */ + 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, /* 110-119 */ + 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ +}; + /* The following diagram shows the logical steps by which three octets get transformed into four base64 characters. @@ -3241,17 +3269,21 @@ #define READ_QUADRUPLET_BYTE(retval) \ base64 characters. */ -static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool); +static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool, + bool, bool); static ptrdiff_t base64_decode_1 (const char *, char *, ptrdiff_t, bool, - ptrdiff_t *); + bool, ptrdiff_t *); DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, - 2, 3, "r", + 2, 5, "r", doc: /* Base64-encode the region between BEG and END. Return the length of the encoded text. Optional third argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break) +into shorter lines. +Optional fourth argument NO-PAD means do not add padding char =. +Optional fifth argument URL-VARIANT means use Url variant (RFC4648). */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object url_variant) { char *encoded; ptrdiff_t allength, length; @@ -3275,6 +3307,7 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 ((char *) BYTE_POS_ADDR (ibeg), encoded, length, NILP (no_line_break), + NILP(no_pad), !NILP(url_variant), !NILP (BVAR (current_buffer, enable_multibyte_characters))); if (encoded_length > allength) emacs_abort (); @@ -3306,11 +3339,14 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, } DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, - 1, 2, 0, + 1, 4, 0, doc: /* Base64-encode STRING and return the result. Optional second argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object string, Lisp_Object no_line_break) +into shorter lines. +Optional third argument NO-PAD means do not add padding char =. +Optional fourth argument URL-VARIANT means use Url variant (RFC4648). */) + (Lisp_Object string, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object url_variant) { ptrdiff_t allength, length, encoded_length; char *encoded; @@ -3331,6 +3367,7 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, encoded_length = base64_encode_1 (SSDATA (string), encoded, length, NILP (no_line_break), + NILP(no_pad), !NILP(url_variant), STRING_MULTIBYTE (string)); if (encoded_length > allength) emacs_abort (); @@ -3349,7 +3386,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, static ptrdiff_t base64_encode_1 (const char *from, char *to, ptrdiff_t length, - bool line_break, bool multibyte) + bool line_break, bool pad, bool url_variant, + bool multibyte) { int counter = 0; ptrdiff_t i = 0; @@ -3357,6 +3395,7 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, int c; unsigned int value; int bytes; + char const *b64_value_to_char = (url_variant) ? base64url_value_to_char : base64_value_to_char; while (i < length) { @@ -3387,16 +3426,18 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, /* Process first byte of a triplet. */ - *e++ = base64_value_to_char[0x3f & c >> 2]; + *e++ = b64_value_to_char[0x3f & c >> 2]; value = (0x03 & c) << 4; /* Process second byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) { + *e++ = '='; + *e++ = '='; + } break; } @@ -3412,15 +3453,17 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x0f & c >> 4)]; + *e++ = b64_value_to_char[value | (0x0f & c >> 4)]; value = (0x0f & c) << 2; /* Process third byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) { + *e++ = '='; + } break; } @@ -3436,8 +3479,8 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x03 & c >> 6)]; - *e++ = base64_value_to_char[0x3f & c]; + *e++ = b64_value_to_char[value | (0x03 & c >> 6)]; + *e++ = b64_value_to_char[0x3f & c]; } return e - to; @@ -3445,11 +3488,13 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, - 2, 2, "r", + 2, 3, "r", doc: /* Base64-decode the region between BEG and END. Return the length of the decoded text. -If the region can't be decoded, signal an error and don't modify the buffer. */) - (Lisp_Object beg, Lisp_Object end) +If the region can't be decoded, signal an error and don't modify the buffer. +Optional third argument URL-VARIANT define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object url_variant) { ptrdiff_t ibeg, iend, length, allength; char *decoded; @@ -3474,7 +3519,7 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, move_gap_both (XFIXNAT (beg), ibeg); decoded_length = base64_decode_1 ((char *) BYTE_POS_ADDR (ibeg), - decoded, length, + decoded, length, !NILP(url_variant), multibyte, &inserted_chars); if (decoded_length > allength) emacs_abort (); @@ -3508,9 +3553,11 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, } DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, - 1, 1, 0, - doc: /* Base64-decode STRING and return the result. */) - (Lisp_Object string) + 1, 2, 0, + doc: /* Base64-decode STRING and return the result +Optional argument URL-VARIANT define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object string, Lisp_Object url_variant) { char *decoded; ptrdiff_t length, decoded_length; @@ -3525,7 +3572,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, /* The decoded result should be unibyte. */ decoded_length = base64_decode_1 (SSDATA (string), decoded, length, - 0, NULL); + !NILP(url_variant), 0, NULL); if (decoded_length > length) emacs_abort (); else if (decoded_length >= 0) @@ -3547,6 +3594,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, static ptrdiff_t base64_decode_1 (const char *from, char *to, ptrdiff_t length, + bool url_variant, bool multibyte, ptrdiff_t *nchars_return) { ptrdiff_t i = 0; /* Used inside READ_QUADRUPLET_BYTE */ @@ -3554,6 +3602,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, unsigned char c; unsigned long value; ptrdiff_t nchars = 0; + short const *b64_char_to_value = (url_variant) ? base64url_char_to_value : base64_char_to_value; while (1) { @@ -3563,7 +3612,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value = base64_char_to_value[c] << 18; + value = b64_char_to_value[c] << 18; /* Process second byte of a quadruplet. */ @@ -3571,7 +3620,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 12; + value |= b64_char_to_value[c] << 12; c = (unsigned char) (value >> 16); if (multibyte && c >= 128) @@ -3582,7 +3631,14 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process third byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!url_variant) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') { @@ -3595,7 +3651,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 6; + value |= b64_char_to_value[c] << 6; c = (unsigned char) (0xff & value >> 8); if (multibyte && c >= 128) @@ -3606,14 +3662,21 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process fourth byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!url_variant) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') continue; if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c]; + value |= b64_char_to_value[c]; c = (unsigned char) (0xff & value); if (multibyte && c >= 128) diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el index 6ebab4287f..3301129567 100644 --- a/test/src/fns-tests.el +++ b/test/src/fns-tests.el @@ -233,6 +233,137 @@ fns-tests-func-arity (should (equal (func-arity (eval (lambda (x &optional y)) t)) '(1 . 2))) (should (equal (func-arity 'let) '(1 . unevalled)))) +(defun string-repeat (s o) + (apply 'concat (make-list o s))) + +(ert-deftest fns-tests-base64-encode-string () + ;; standard variant RFC2045 + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f") "Zg==")) + (should (equal (base64-encode-string "fo") "Zm8=")) + (should (equal (base64-encode-string "foo") "Zm9v")) + (should (equal (base64-encode-string "foob") "Zm9vYg==")) + (should (equal (base64-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/")) + + ;; no line break + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string (string-repeat "f" 100) t) (concat (string-repeat "Zm" 66) "Zg=="))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw=="))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) (string-repeat "FPucA9l+" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) (string-repeat "FPucA9l/" 10))) + + ;; no paddign + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f" nil t) "Zg")) + (should (equal (base64-encode-string "fo" nil t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t) "Zm9vYmFy")) + + ;; url variant wih padding + (should (equal (base64-encode-string "" nil nil t) "")) + (should (equal (base64-encode-string "f" nil nil t) "Zg==")) + (should (equal (base64-encode-string "fo" nil nil t) "Zm8=")) + (should (equal (base64-encode-string "foo" nil nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil nil t) "Zm9vYg==")) + (should (equal (base64-encode-string "fooba" nil nil t) "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar" nil nil t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil nil t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil nil t) "FPucA9l_")) + + ;; url variant no padding + (should (equal (base64-encode-string "" nil t t) "")) + (should (equal (base64-encode-string "f" nil t t) "Zg")) + (should (equal (base64-encode-string "fo" nil t t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil t t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil t t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (base64-encode-string (string-repeat "f" 100) t t t) (concat (string-repeat "Zm" 66) "Zg"))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t t t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t t t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t t t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t t t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t t t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t t t) (string-repeat "FPucA9l-" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t t t) (string-repeat "FPucA9l_" 10))) + + ) + +(ert-deftest fns-tests-base64-decode-string () + ;; standard variant RFC2045 + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==") "f")) + (should (equal (base64-decode-string "Zm8=") "fo")) + (should (equal (base64-decode-string "Zm9v") "foo")) + (should (equal (base64-decode-string "Zm9vYg==") "foob")) + (should (equal (base64-decode-string "Zm9vYmE=") "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy") "foobar")) + (should (equal (base64-decode-string "FPucA9l+") "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l/") "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; no paddign + (should (equal (base64-decode-string "" t) "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + + ;; url variant wih padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==" t) "f") ) + (should (equal (base64-decode-string "Zm8=" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg==" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE=" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; url variant no padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + + ;; url variant no line break no padding + (should (equal (base64-decode-string (concat (string-repeat "Zm" 66) "Zg") t) (string-repeat "f" 100))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw") t) (string-repeat "fo" 50))) + (should (equal (base64-decode-string (string-repeat "Zm9v" 25) t) (string-repeat "foo" 25))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmZvb2Jmb29i" 5) t) (string-repeat "foob" 15))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5) t) (string-repeat "fooba" 15))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy") t) (string-repeat "foobar" 15))) + (should (equal (base64-decode-string (string-repeat "FPucA9l-" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10))) + (should (equal (base64-decode-string (string-repeat "FPucA9l_" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10))) + + ;; errors check + (should (eq :got-error (condition-case () (base64-decode-string "Zg=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmE") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmFy=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zg=Zg=") (error :got-error)))) + ) + (ert-deftest fns-tests-hash-buffer () (should (equal (sha1 "foo") "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33")) (should (equal (with-temp-buffer -- 2.21.0 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-21 22:32 Add support for base64url variant Pierre Téchoueyres @ 2019-05-22 7:43 ` Eli Zaretskii 2019-05-22 9:25 ` Pierre Téchoueyres 2019-05-23 18:50 ` Eli Zaretskii 1 sibling, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-05-22 7:43 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: Pierre Téchoueyres <pierre.techoueyres@free.fr> > Date: Wed, 22 May 2019 00:32:19 +0200 > > I'm actually working on an emacs library to work with JWT (JSON Web > Token) as described in RFC 7519 / 7797. To achive that I must use an > variant of base 64 encoding named base64url as described in RFC4648. > > Actually Emacs doesn't provide this variant. I've attached an attempt > to implement it at C level. > > I'm avare that my patch is far from perfect and I would welcome any > help or comments to improve it. Thanks, but please avoid posting such large patches as long as you don't have a copyright assignment for Emacs on file. That's because just by reading the patch, people will remember some of the implementation, and will then have hard time to come up with a different one, should your copyright assignment paperwork not come through. If you'd like to sign the forms, I will send them to you. Then we could revisit the patch when the legal paperwork is done. Thanks again for working on this. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-22 7:43 ` Eli Zaretskii @ 2019-05-22 9:25 ` Pierre Téchoueyres 2019-05-22 9:50 ` Eli Zaretskii 0 siblings, 1 reply; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-22 9:25 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel Hello Eli, > > Thanks, but please avoid posting such large patches as long as you > don't have a copyright assignment for Emacs on file. I have already signed paperwork for TRAMP and, I think, for Emacs. But maybe there is something wrong with them ? > That's because > just by reading the patch, people will remember some of the > implementation, and will then have hard time to come up with a > different one, should your copyright assignment paperwork not come > through. > > If you'd like to sign the forms, I will send them to you. Then we > could revisit the patch when the legal paperwork is done. If you can't find my previous assignment, then yes send me them back. > Thanks again for working on this. Hope this could help. I would like to have some discussion on how to improve some points : - Is adding parameter to existing functions the way to go or is it better to add new ones for base64url - I would like to improve base64-decode* in a way it could detect the variant but actually don't know how to do that. - The documentations is not really what I would like to see (and there my English is faulty) Thanks in advance. Pierre. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-22 9:25 ` Pierre Téchoueyres @ 2019-05-22 9:50 ` Eli Zaretskii 2019-05-23 17:51 ` Pierre Téchoueyres 0 siblings, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-05-22 9:50 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > Date: Wed, 22 May 2019 11:25:34 +0200 > From: Pierre Téchoueyres <pierre.techoueyres@free.fr> > Cc: emacs-devel@gnu.org > > Hello Eli, > > > > Thanks, but please avoid posting such large patches as long as you > > don't have a copyright assignment for Emacs on file. > > I have already signed paperwork for TRAMP and, I think, for Emacs. > But maybe there is something wrong with them ? I do see the assignment for Tramp, but not for Emacs. > If you can't find my previous assignment, then yes send me them back. Will send off-list. > I would like to have some discussion on how to improve some points : > - Is adding parameter to existing functions the way to go or is it > better to add new ones for base64url I think an optional argument will be fine. > - I would like to improve base64-decode* in a way it could detect the > variant but actually don't know how to do that. Maybe someone else will have an idea. Is such an algorithmic detection described someplace? > - The documentations is not really what I would like to see (and there > my English is faulty) Don't worry about that, we will polish it when the time comes. Thanks for providing the documentation in the first place. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-22 9:50 ` Eli Zaretskii @ 2019-05-23 17:51 ` Pierre Téchoueyres 2019-05-23 18:45 ` Noam Postavsky 2019-05-23 18:58 ` Eli Zaretskii 0 siblings, 2 replies; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-23 17:51 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> ... >> I would like to have some discussion on how to improve some points : >> - Is adding parameter to existing functions the way to go or is it >> better to add new ones for base64url > > I think an optional argument will be fine. Except here I've added two, and one that doesn't really mean anything if the second isn't set. ie. you should not (at least without breaking the RFC) generate an base64 string without padding. At first many parameters seemed to me a good thing : limitted patch and flexibility. But now I'm no more sure ... >> - I would like to improve base64-decode* in a way it could detect the >> variant but actually don't know how to do that. > > Maybe someone else will have an idea. Is such an algorithmic > detection described someplace? > None I'm aware of. I was thinking to something like that : - define two boolean. One saying you're on crude base64 another saying you're on base64url variant. - initialize them as false. - start decoding. - when finding crude base64 chars (/ or +) set the base64 to true, - when finding specific url variant chars (- or _) set base64url to true, try to decode until the end of data. On parts where padding is checked do it only if base64 is true. But this approch could fail on the following cases : - a mix of chars from both variant without padding (no checks but obviously wrong) - absence of chars from any variant (here I can't decide for the necessity of padding). I would also bring your attention on the part where I dynamically assign pointers on specialized arrays for encoding (resp. decoding). ex: line 244 of patch char const *b64_value_to_char = (url_variant) ? base64url_value_to_char : base64_value_to_char; Before my change there were static const, so I suppose compiler could have inlined them or at least stored on some cache. But now ... So I'm a little scarried by the possible lost of performance. If anyone has some hint on how I could benchmark this (other than by the naive way which could result in my data where all in cache ...) Pierre. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 17:51 ` Pierre Téchoueyres @ 2019-05-23 18:45 ` Noam Postavsky 2019-05-23 19:32 ` Pierre Téchoueyres 2019-05-23 18:58 ` Eli Zaretskii 1 sibling, 1 reply; 23+ messages in thread From: Noam Postavsky @ 2019-05-23 18:45 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: Eli Zaretskii, Emacs developers On Thu, 23 May 2019 at 13:51, Pierre Téchoueyres <pierre.techoueyres@free.fr> wrote: > > Eli Zaretskii <eliz@gnu.org> writes: > > >> ... > >> I would like to have some discussion on how to improve some points : > >> - Is adding parameter to existing functions the way to go or is it > >> better to add new ones for base64url > > > > I think an optional argument will be fine. > > Except here I've added two, and one that doesn't really mean anything if > the second isn't set. ie. you should not (at least without breaking the > RFC) generate an base64 string without padding. At first many > parameters seemed to me a good thing : limitted patch and flexibility. > But now I'm no more sure ... You could leave out the NO-PAD argument, it's easy enough for the calling Lisp code to delete a couple of "=" chars if really needed. > I would also bring your attention on the part where I dynamically assign > pointers on specialized arrays for encoding (resp. decoding). > > ex: line 244 of patch > > char const *b64_value_to_char = (url_variant) ? base64url_value_to_char : base64_value_to_char; > > > Before my change there were static const, so I suppose compiler could > have inlined them or at least stored on some cache. But now ... > So I'm a little scarried by the possible lost of performance. If anyone > has some hint on how I could benchmark this I doubt it will have a measurable impact on performance. And even if it did have some very tiny effect, what would you do about it? Duplicate the whole encoding function, just to avoid a dynamic choice of array? The difference would have to be pretty drastic to be worth that, IMO. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 18:45 ` Noam Postavsky @ 2019-05-23 19:32 ` Pierre Téchoueyres 0 siblings, 0 replies; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-23 19:32 UTC (permalink / raw) To: Noam Postavsky; +Cc: Eli Zaretskii, Emacs developers Noam Postavsky <npostavs@gmail.com> writes: > On Thu, 23 May 2019 at 13:51, Pierre Téchoueyres > <pierre.techoueyres@free.fr> wrote: >> >> Eli Zaretskii <eliz@gnu.org> writes: >> >> >> ... >> >> I would like to have some discussion on how to improve some points : >> >> - Is adding parameter to existing functions the way to go or is it >> >> better to add new ones for base64url >> > >> > I think an optional argument will be fine. >> >> Except here I've added two, and one that doesn't really mean anything if >> the second isn't set. ie. you should not (at least without breaking the >> RFC) generate an base64 string without padding. At first many >> parameters seemed to me a good thing : limitted patch and flexibility. >> But now I'm no more sure ... > > You could leave out the NO-PAD argument, it's easy enough for the > calling Lisp code to delete a couple of "=" chars if really needed. > This could be an option ... but I do see this as ugly adding characters on one side to remove them from the other. Maybe it could be better to split the functions then ? What about (base64-encode-string STRING &optional NO-LINE-BREAK) (base64url-encode-string STRING &optional NO-PAD) and (base64-encode-region BEG END &optional NO-LINE-BREAK) (base64url-encode-region BEG END &optional NO-PAD) >> I would also bring your attention on the part where I dynamically assign >> pointers on specialized arrays for encoding (resp. decoding). >> >> ex: line 244 of patch >> >> char const *b64_value_to_char = (url_variant) ? base64url_value_to_char : base64_value_to_char; >> >> >> Before my change there were static const, so I suppose compiler could >> have inlined them or at least stored on some cache. But now ... >> So I'm a little scarried by the possible lost of performance. If anyone >> has some hint on how I could benchmark this > > I doubt it will have a measurable impact on performance. And even if > it did have some very tiny effect, what would you do about it? > Duplicate the whole encoding function, just to avoid a dynamic choice > of array? The difference would have to be pretty drastic to be worth > that, IMO. Thanks, I hoped for something like your answer. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 17:51 ` Pierre Téchoueyres 2019-05-23 18:45 ` Noam Postavsky @ 2019-05-23 18:58 ` Eli Zaretskii 1 sibling, 0 replies; 23+ messages in thread From: Eli Zaretskii @ 2019-05-23 18:58 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: pierre.techoueyres@free.fr (Pierre Téchoueyres) > Cc: emacs-devel@gnu.org > Date: Thu, 23 May 2019 19:51:00 +0200 > > > I think an optional argument will be fine. > > Except here I've added two, and one that doesn't really mean anything if > the second isn't set. Then perhaps switch their order. > >> - I would like to improve base64-decode* in a way it could detect the > >> variant but actually don't know how to do that. > > > > Maybe someone else will have an idea. Is such an algorithmic > > detection described someplace? > > > None I'm aware of. I was thinking to something like that : > - define two boolean. One saying you're on crude base64 another saying > you're on base64url variant. > - initialize them as false. > - start decoding. > - when finding crude base64 chars (/ or +) set the base64 to true, > - when finding specific url variant chars (- or _) set base64url to true, Doesn't sound worth it. Just an argument passed by the caller should be fine. The caller will have to know how the text is encoded. > I would also bring your attention on the part where I dynamically assign > pointers on specialized arrays for encoding (resp. decoding). I don't see a problem with that. > Before my change there were static const, so I suppose compiler could > have inlined them or at least stored on some cache. But now ... > So I'm a little scarried by the possible lost of performance. If anyone > has some hint on how I could benchmark this (other than by the naive way > which could result in my data where all in cache ...) Emacs has benchmarking facilities, so you could time the code and see if there's some performance penalty. If the performance suffers too much, we could always have a helper function which receives the array as a parameter. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-21 22:32 Add support for base64url variant Pierre Téchoueyres 2019-05-22 7:43 ` Eli Zaretskii @ 2019-05-23 18:50 ` Eli Zaretskii 2019-05-23 19:37 ` Pierre Téchoueyres 1 sibling, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-05-23 18:50 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: Pierre Téchoueyres <pierre.techoueyres@free.fr> > Date: Wed, 22 May 2019 00:32:19 +0200 > > + NILP(no_pad), !NILP(url_variant), Please leave one blank between the name of a macro or function and the following opening parenthesis. > + if (pad) { > + *e++ = '='; > + *e++ = '='; > + } This is not our style of writing blocks in braces. We use this style: if (pad) { *e++ = '='; ... > + *e++ = b64_value_to_char[value]; > + if (pad) { > + *e++ = '='; > + } Likewise. I'd suggest to call the new argument base64url or somesuch, since this is trhe official name. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 18:50 ` Eli Zaretskii @ 2019-05-23 19:37 ` Pierre Téchoueyres 2019-05-23 19:51 ` Eli Zaretskii 0 siblings, 1 reply; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-23 19:37 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> From: Pierre Téchoueyres <pierre.techoueyres@free.fr> >> Date: Wed, 22 May 2019 00:32:19 +0200 >> >> + NILP(no_pad), !NILP(url_variant), > > Please leave one blank between the name of a macro or function and the > following opening parenthesis. Fixed. > >> + if (pad) { >> + *e++ = '='; >> + *e++ = '='; >> + } > > This is not our style of writing blocks in braces. We use this style: > > if (pad) > { > *e++ = '='; > ... > >> + *e++ = b64_value_to_char[value]; >> + if (pad) { >> + *e++ = '='; >> + } > > Likewise. > Fixed. > I'd suggest to call the new argument base64url or somesuch, since > this is trhe official name. > You mean in replacement of url_variant or b64_value_to_char ? How should I send new versions of the patch ? As a full patch in attachment like previously ? ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 19:37 ` Pierre Téchoueyres @ 2019-05-23 19:51 ` Eli Zaretskii 2019-05-27 20:30 ` Pierre Téchoueyres 0 siblings, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-05-23 19:51 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: pierre.techoueyres@free.fr (Pierre Téchoueyres) > Cc: emacs-devel@gnu.org > Date: Thu, 23 May 2019 21:37:01 +0200 > > > I'd suggest to call the new argument base64url or somesuch, since > > this is trhe official name. > > > You mean in replacement of url_variant or b64_value_to_char ? The former. > How should I send new versions of the patch ? As a full patch in > attachment like previously ? Yes. And please include the log messages (in the ChangeLog style). ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-23 19:51 ` Eli Zaretskii @ 2019-05-27 20:30 ` Pierre Téchoueyres 2019-06-07 21:04 ` Pierre Téchoueyres 2019-06-08 8:18 ` Eli Zaretskii 0 siblings, 2 replies; 23+ messages in thread From: Pierre Téchoueyres @ 2019-05-27 20:30 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel [-- Attachment #1: Type: text/plain, Size: 663 bytes --] Eli Zaretskii <eliz@gnu.org> writes: >> From: pierre.techoueyres@free.fr (Pierre Téchoueyres) >> Cc: emacs-devel@gnu.org >> Date: Thu, 23 May 2019 21:37:01 +0200 >> >> > I'd suggest to call the new argument base64url or somesuch, since >> > this is trhe official name. >> > >> You mean in replacement of url_variant or b64_value_to_char ? > > The former. > >> How should I send new versions of the patch ? As a full patch in >> attachment like previously ? > > Yes. And please include the log messages (in the ChangeLog style). > > You'll find two patches attached. First one contains requested changes and, hoppefully, a valid Changelog [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: with parameters --] [-- Type: text/x-patch, Size: 24899 bytes --] From dcd03d67d4d7fa9d1f46554a6703ae562ef39ce5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr> Date: Tue, 21 May 2019 23:00:13 +0200 Subject: [PATCH] Add support for base64url variant. Implement the RFC4648 variant of base64 encoding used with url. * doc/lispref/text.texi (base64-encode-region, base64-encode-string, base64-decode-region, base64-decode-string): Adds notice for optionals parameters. * etc/NEWS: Announce support for URL variant of base 64 functions. * src/fns.c (base64-encode-region, base64-encode-string) : Adds optional parameters to manage padding and url variant. (base64-decode-region, base64-decode-string): Adds optional parameter to indicate use of url-variant. (base64-encode-1): Adds parameters to manage padding and url variant. (base64-decode-1): Adds parameter to manage url variant. * test/src/fns-tests.el: Adds tests for encoding / decoding variants. --- doc/lispref/text.texi | 29 ++++++++-- etc/NEWS | 7 +++ src/fns.c | 131 +++++++++++++++++++++++++++++++----------- test/src/fns-tests.el | 131 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 260 insertions(+), 38 deletions(-) diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 278bc3c268..ca1bc4e2df 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4541,10 +4541,10 @@ Base 64 usually written by technical experts acting on their own initiative, and are traditionally written in a pragmatic, experience-driven manner. -}2045. This section describes the functions for +}2045 and also in RFC4648. This section describes the functions for converting to and from this code. -@deffn Command base64-encode-region beg end &optional no-line-break +@deffn Command base64-encode-region beg end &optional no-line-break no-pad base64url This function converts the region from @var{beg} to @var{end} into base 64 code. It returns the length of the encoded text. An error is signaled if a character in the region is multibyte, i.e., in a @@ -4556,9 +4556,15 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the output is just one long line. + +If optional argument @var{no-pad} is set then padding isn't +generated. + +If optional argument @var{base64url} is set, then chars @code{+} and +@code{/} are replaced by @code{-} and @code{_} (See RFC4648). @end deffn -@defun base64-encode-string string &optional no-line-break +@defun base64-encode-string string &optional no-line-break no-pad base64url This function converts the string @var{string} into base 64 code. It returns a string containing the encoded text. As for @code{base64-encode-region}, an error is signaled if a character in the @@ -4568,22 +4574,35 @@ Base 64 text, to avoid overlong lines. However, if the optional argument @var{no-line-break} is non-@code{nil}, these newlines are not added, so the result string is just one long line. + +If optional argument @var{no-pad} is non-@code{nil} then padding isn't +generated. + +If optional argument @var{base64url} is non-@code{nil} then chars +@code{+} and @code{/} are replaced by @code{-} abd @code{_} (See +RFC4648). @end defun -@deffn Command base64-decode-region beg end +@deffn Command base64-decode-region beg end &optional base64url This function converts the region from @var{beg} to @var{end} from base 64 code into the corresponding decoded text. It returns the length of the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is non-@code{nil} then padding +become optionnal and url variant is used (See RFC4648). @end deffn -@defun base64-decode-string string +@defun base64-decode-string string &optional base64url This function converts the string @var{string} from base 64 code into the corresponding decoded text. It returns a unibyte string containing the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is non-@code{nil} then padding become +optionnal and url variant is used (See RFC4648). @end defun @node Checksum/Hash diff --git a/etc/NEWS b/etc/NEWS index 222b86ee2b..9d7bfe5e82 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -370,6 +370,13 @@ in tooltips, as it is not useful there. There are 2 new buffer local variables and 1 face to customize this mode they are described in the manual "(emacs) Display". ++++ +** Functions 'base64-(encode|decode)-(string|region)' now manage url variant (RFC4648) +The functions 'base64-encode-(region|string)' now have optionals +arguments to mange padding and url variant. +Mirror function 'base64-decode-(region|string)' now have an optional +argument url-variant to manage this RFC. + \f * Editing Changes in Emacs 27.1 diff --git a/src/fns.c b/src/fns.c index 6b1f7331f5..963b54fb86 100644 --- a/src/fns.c +++ b/src/fns.c @@ -3169,7 +3169,7 @@ #define MIME_LINE_LENGTH 76 #define IS_ASCII(Character) \ ((Character) < 128) #define IS_BASE64(Character) \ - (IS_ASCII (Character) && base64_char_to_value[Character] >= 0) + (IS_ASCII (Character) && b64_char_to_value[Character] >= 0) #define IS_BASE64_IGNORABLE(Character) \ ((Character) == ' ' || (Character) == '\t' || (Character) == '\n' \ || (Character) == '\f' || (Character) == '\r') @@ -3202,6 +3202,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ '8', '9', '+', '/' /* 60-63 */ }; +static const char base64url_value_to_char[64] = +{ + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', /* 0- 9 */ + 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', /* 10-19 */ + 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', /* 20-29 */ + 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', /* 30-39 */ + 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', /* 40-49 */ + 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', /* 50-59 */ + '8', '9', '-', '_' /* 60-63 */ +}; + /* Table of base64 values for first 128 characters. */ static const short base64_char_to_value[128] = { @@ -3220,6 +3231,23 @@ #define READ_QUADRUPLET_BYTE(retval) \ 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ }; +static const short base64url_char_to_value[128] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 0- 9 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 10- 19 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 20- 29 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 30- 39 */ + -1, -1, -1, -1, -1, 62, -1, -1, 52, 53, /* 40- 49 */ + 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, /* 50- 59 */ + -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, /* 60- 69 */ + 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 70- 79 */ + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, /* 80- 89 */ + 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, /* 90- 99 */ + 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, /* 100-109 */ + 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, /* 110-119 */ + 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ +}; + /* The following diagram shows the logical steps by which three octets get transformed into four base64 characters. @@ -3239,17 +3267,21 @@ #define READ_QUADRUPLET_BYTE(retval) \ base64 characters. */ -static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool); +static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool, + bool, bool); static ptrdiff_t base64_decode_1 (const char *, char *, ptrdiff_t, bool, - ptrdiff_t *); + bool, ptrdiff_t *); DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, - 2, 3, "r", + 2, 5, "r", doc: /* Base64-encode the region between BEG and END. Return the length of the encoded text. Optional third argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break) +into shorter lines. +Optional fourth argument NO-PAD means do not add padding char =. +Optional fifth argument URL-VARIANT means use Url variant (RFC4648). */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object base64url) { char *encoded; ptrdiff_t allength, length; @@ -3273,6 +3305,7 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 ((char *) BYTE_POS_ADDR (ibeg), encoded, length, NILP (no_line_break), + NILP (no_pad), !NILP (base64url), !NILP (BVAR (current_buffer, enable_multibyte_characters))); if (encoded_length > allength) emacs_abort (); @@ -3304,11 +3337,14 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, } DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, - 1, 2, 0, + 1, 4, 0, doc: /* Base64-encode STRING and return the result. Optional second argument NO-LINE-BREAK means do not break long lines -into shorter lines. */) - (Lisp_Object string, Lisp_Object no_line_break) +into shorter lines. +Optional third argument NO-PAD means do not add padding char =. +Optional fourth argument BASE64URL means use Url variant (RFC4648). */) + (Lisp_Object string, Lisp_Object no_line_break, + Lisp_Object no_pad, Lisp_Object base64url) { ptrdiff_t allength, length, encoded_length; char *encoded; @@ -3329,6 +3365,7 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, encoded_length = base64_encode_1 (SSDATA (string), encoded, length, NILP (no_line_break), + NILP (no_pad), !NILP (base64url), STRING_MULTIBYTE (string)); if (encoded_length > allength) emacs_abort (); @@ -3347,7 +3384,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, static ptrdiff_t base64_encode_1 (const char *from, char *to, ptrdiff_t length, - bool line_break, bool multibyte) + bool line_break, bool pad, bool base64url, + bool multibyte) { int counter = 0; ptrdiff_t i = 0; @@ -3355,6 +3393,7 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, int c; unsigned int value; int bytes; + char const *b64_value_to_char = (base64url) ? base64url_value_to_char : base64_value_to_char; while (i < length) { @@ -3385,16 +3424,19 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, /* Process first byte of a triplet. */ - *e++ = base64_value_to_char[0x3f & c >> 2]; + *e++ = b64_value_to_char[0x3f & c >> 2]; value = (0x03 & c) << 4; /* Process second byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + *e++ = '='; + } break; } @@ -3410,15 +3452,18 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x0f & c >> 4)]; + *e++ = b64_value_to_char[value | (0x0f & c >> 4)]; value = (0x0f & c) << 2; /* Process third byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + } break; } @@ -3434,8 +3479,8 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x03 & c >> 6)]; - *e++ = base64_value_to_char[0x3f & c]; + *e++ = b64_value_to_char[value | (0x03 & c >> 6)]; + *e++ = b64_value_to_char[0x3f & c]; } return e - to; @@ -3443,11 +3488,13 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, - 2, 2, "r", + 2, 3, "r", doc: /* Base64-decode the region between BEG and END. Return the length of the decoded text. -If the region can't be decoded, signal an error and don't modify the buffer. */) - (Lisp_Object beg, Lisp_Object end) +If the region can't be decoded, signal an error and don't modify the buffer. +Optional third argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object base64url) { ptrdiff_t ibeg, iend, length, allength; char *decoded; @@ -3472,7 +3519,7 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, move_gap_both (XFIXNAT (beg), ibeg); decoded_length = base64_decode_1 ((char *) BYTE_POS_ADDR (ibeg), - decoded, length, + decoded, length, !NILP (base64url), multibyte, &inserted_chars); if (decoded_length > allength) emacs_abort (); @@ -3506,9 +3553,11 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, } DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, - 1, 1, 0, - doc: /* Base64-decode STRING and return the result. */) - (Lisp_Object string) + 1, 2, 0, + doc: /* Base64-decode STRING and return the result +Optional argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object string, Lisp_Object base64url) { char *decoded; ptrdiff_t length, decoded_length; @@ -3523,7 +3572,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, /* The decoded result should be unibyte. */ decoded_length = base64_decode_1 (SSDATA (string), decoded, length, - 0, NULL); + !NILP (base64url), 0, NULL); if (decoded_length > length) emacs_abort (); else if (decoded_length >= 0) @@ -3545,6 +3594,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, static ptrdiff_t base64_decode_1 (const char *from, char *to, ptrdiff_t length, + bool base64url, bool multibyte, ptrdiff_t *nchars_return) { ptrdiff_t i = 0; /* Used inside READ_QUADRUPLET_BYTE */ @@ -3552,6 +3602,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, unsigned char c; unsigned long value; ptrdiff_t nchars = 0; + short const *b64_char_to_value = (base64url) ? base64url_char_to_value : base64_char_to_value; while (1) { @@ -3561,7 +3612,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value = base64_char_to_value[c] << 18; + value = b64_char_to_value[c] << 18; /* Process second byte of a quadruplet. */ @@ -3569,7 +3620,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 12; + value |= b64_char_to_value[c] << 12; c = (unsigned char) (value >> 16); if (multibyte && c >= 128) @@ -3580,7 +3631,14 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process third byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') { @@ -3593,7 +3651,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 6; + value |= b64_char_to_value[c] << 6; c = (unsigned char) (0xff & value >> 8); if (multibyte && c >= 128) @@ -3604,14 +3662,21 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process fourth byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') continue; if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c]; + value |= b64_char_to_value[c]; c = (unsigned char) (0xff & value); if (multibyte && c >= 128) diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el index 6ebab4287f..3301129567 100644 --- a/test/src/fns-tests.el +++ b/test/src/fns-tests.el @@ -233,6 +233,137 @@ fns-tests-func-arity (should (equal (func-arity (eval (lambda (x &optional y)) t)) '(1 . 2))) (should (equal (func-arity 'let) '(1 . unevalled)))) +(defun string-repeat (s o) + (apply 'concat (make-list o s))) + +(ert-deftest fns-tests-base64-encode-string () + ;; standard variant RFC2045 + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f") "Zg==")) + (should (equal (base64-encode-string "fo") "Zm8=")) + (should (equal (base64-encode-string "foo") "Zm9v")) + (should (equal (base64-encode-string "foob") "Zm9vYg==")) + (should (equal (base64-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/")) + + ;; no line break + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string (string-repeat "f" 100) t) (concat (string-repeat "Zm" 66) "Zg=="))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw=="))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) (string-repeat "FPucA9l+" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) (string-repeat "FPucA9l/" 10))) + + ;; no paddign + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f" nil t) "Zg")) + (should (equal (base64-encode-string "fo" nil t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t) "Zm9vYmFy")) + + ;; url variant wih padding + (should (equal (base64-encode-string "" nil nil t) "")) + (should (equal (base64-encode-string "f" nil nil t) "Zg==")) + (should (equal (base64-encode-string "fo" nil nil t) "Zm8=")) + (should (equal (base64-encode-string "foo" nil nil t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil nil t) "Zm9vYg==")) + (should (equal (base64-encode-string "fooba" nil nil t) "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar" nil nil t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil nil t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil nil t) "FPucA9l_")) + + ;; url variant no padding + (should (equal (base64-encode-string "" nil t t) "")) + (should (equal (base64-encode-string "f" nil t t) "Zg")) + (should (equal (base64-encode-string "fo" nil t t) "Zm8")) + (should (equal (base64-encode-string "foo" nil t t) "Zm9v")) + (should (equal (base64-encode-string "foob" nil t t) "Zm9vYg")) + (should (equal (base64-encode-string "fooba" nil t t) "Zm9vYmE")) + (should (equal (base64-encode-string "foobar" nil t t) "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e" nil t t) "FPucA9l-")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f" nil t t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (base64-encode-string (string-repeat "f" 100) t t t) (concat (string-repeat "Zm" 66) "Zg"))) + (should (equal (base64-encode-string (string-repeat "fo" 50) t t t) (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (base64-encode-string (string-repeat "foo" 25) t t t) (string-repeat "Zm9v" 25))) + (should (equal (base64-encode-string (string-repeat "foob" 15) t t t) (string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64-encode-string (string-repeat "fooba" 15) t t t) (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64-encode-string (string-repeat "foobar" 15) t t t) (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t t t) (string-repeat "FPucA9l-" 10))) + (should (equal (base64-encode-string (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t t t) (string-repeat "FPucA9l_" 10))) + + ) + +(ert-deftest fns-tests-base64-decode-string () + ;; standard variant RFC2045 + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==") "f")) + (should (equal (base64-decode-string "Zm8=") "fo")) + (should (equal (base64-decode-string "Zm9v") "foo")) + (should (equal (base64-decode-string "Zm9vYg==") "foob")) + (should (equal (base64-decode-string "Zm9vYmE=") "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy") "foobar")) + (should (equal (base64-decode-string "FPucA9l+") "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l/") "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; no paddign + (should (equal (base64-decode-string "" t) "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + + ;; url variant wih padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==" t) "f") ) + (should (equal (base64-decode-string "Zm8=" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg==" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE=" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; url variant no padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + + ;; url variant no line break no padding + (should (equal (base64-decode-string (concat (string-repeat "Zm" 66) "Zg") t) (string-repeat "f" 100))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9mb2Zv" 16) "Zm9mbw") t) (string-repeat "fo" 50))) + (should (equal (base64-decode-string (string-repeat "Zm9v" 25) t) (string-repeat "foo" 25))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmZvb2Jmb29i" 5) t) (string-repeat "foob" 15))) + (should (equal (base64-decode-string (string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5) t) (string-repeat "fooba" 15))) + (should (equal (base64-decode-string (concat (string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy") t) (string-repeat "foobar" 15))) + (should (equal (base64-decode-string (string-repeat "FPucA9l-" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10))) + (should (equal (base64-decode-string (string-repeat "FPucA9l_" 10) t) (string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10))) + + ;; errors check + (should (eq :got-error (condition-case () (base64-decode-string "Zg=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmE") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmFy=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zg=Zg=") (error :got-error)))) + ) + (ert-deftest fns-tests-hash-buffer () (should (equal (sha1 "foo") "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33")) (should (equal (with-temp-buffer -- 2.21.0 [-- Attachment #3: Type: text/plain, Size: 281 bytes --] The second patch contains an reworked version wich instead of adding many parameters to base64-encode-region (resp. base64-encode-string) function create the base64url-encode-region (resp. base64url-encode-string) function. Documentation and tests are also updated consequently. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: new functions --] [-- Type: text/x-patch, Size: 30190 bytes --] From 8acbb905aeb49fb0bfb747acc6b4965cf120e23b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr> Date: Tue, 21 May 2019 23:00:13 +0200 Subject: [PATCH] Add support for base64url variant. Implements the RFC4648 variant of base64 encoding used by URLs. * doc/lispref/text.texi (base64url-encode-region, base64url-encode-string): Documents new functions. (base64-decode-region, base64-decode-string): Documents new optional parameter 'base64url' used to use url variant when decoding data. * etc/NEWS: Announce new functions and optionals parameters. * src/fns.c (base64url-encode-region, base64url-encode-region): New functions to manage url variant. (base64-decode-region, base64-decode-string): Adds optional parameter to indicate use of url-variant. (base64_encode_region_1, base64_encode_string_1): Internal functions with extracted code from 'base64_encode_region' and 'base64_encode_string' and optional parameters to manage padding and url variant. (base64-encode-region, base64-encode-string) : Use internal functions previously defined. (base64-encode-1): Adds parameters to manage padding and url variant. (base64-decode-1): Adds parameter to manage url variant. * test/src/fns-tests.el (fns-tests--with-region): New helper macro to test region variant of base64 encode / decode functions. (fns-tests--string-repeat): Helper function used in base64 tests. (fns-tests-base64-encode-region, fns-tests-base64-encode-string): Tests for standard base64 function. (fns-test-base64url-encode-region, fns-test-base64url-encode-string): Tests for url variant. (fns-tests-base64-decode-string): Tests for decoding part. --- doc/lispref/text.texi | 42 +++++++++- etc/NEWS | 8 ++ src/fns.c | 166 ++++++++++++++++++++++++++++++++------- test/src/fns-tests.el | 179 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 363 insertions(+), 32 deletions(-) diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 278bc3c268..3e0cf4c06f 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4541,7 +4541,7 @@ Base 64 usually written by technical experts acting on their own initiative, and are traditionally written in a pragmatic, experience-driven manner. -}2045. This section describes the functions for +}2045 and also in RFC4648. This section describes the functions for converting to and from this code. @deffn Command base64-encode-region beg end &optional no-line-break @@ -4558,6 +4558,22 @@ Base 64 the output is just one long line. @end deffn +@deffn Command base64url-encode-region beg end &optional no-pad +This function converts the region from @var{beg} to @var{end} into base +64 code. It returns the length of the encoded text. An error is +signaled if a character in the region is multibyte, i.e., in a +multibyte buffer the region must contain only characters from the +charsets @code{ascii}, @code{eight-bit-control} and +@code{eight-bit-graphic}. + +Contrary to the function @code{base64-encode-region}, this function +doesnt inserts newline characters into the encoded text, so the output +is just one long line. + +If the optional argument @var{no-pad} is non-@code{nil} then padding +(@code{=}) isn't generated. +@end deffn + @defun base64-encode-string string &optional no-line-break This function converts the string @var{string} into base 64 code. It returns a string containing the encoded text. As for @@ -4570,20 +4586,40 @@ Base 64 the result string is just one long line. @end defun -@deffn Command base64-decode-region beg end +@defun base64url-encode-string string &optional no-pad +This function converts the string @var{string} into base 64 url code +(see RFC4648). It returns a string containing the encoded text. As +for @code{base64url-encode-region}, an error is signaled if a +character in the string is multibyte. + +Contrary to @code{base64-encode-string}, this function doesnt inserts +newline characters into the encoded text, so the result string is just +one long line. + +If the optional argument @var{no-pad} is non-@code{nil} then padding +(@code{=}) isn't generated. +@end defun + +@deffn Command base64-decode-region beg end &optional base64url This function converts the region from @var{beg} to @var{end} from base 64 code into the corresponding decoded text. It returns the length of the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is is non-@code{nil} then padding +become optionnal and url variant is used (see RFC4648). @end deffn -@defun base64-decode-string string +@defun base64-decode-string string &optional base64url This function converts the string @var{string} from base 64 code into the corresponding decoded text. It returns a unibyte string containing the decoded text. The decoding functions ignore newline characters in the encoded text. + +If optional argument @var{base64url} is is non-@code{nil} then padding +become optionnal and url variant is used (see RFC4648). @end defun @node Checksum/Hash diff --git a/etc/NEWS b/etc/NEWS index 222b86ee2b..cae6b25aca 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -370,6 +370,14 @@ in tooltips, as it is not useful there. There are 2 new buffer local variables and 1 face to customize this mode they are described in the manual "(emacs) Display". ++++ +** New functions 'base64url-encode-(string|region)' manage url variant (RFC4648) +The new functions 'base64url-encode-region' and +'base64url-encode-region' now implements url-variant as defined in RFC. + +Mirror functions 'base64-decode-(region|string)' now have an optional +argument base64url to manage this RFC. + \f * Editing Changes in Emacs 27.1 diff --git a/src/fns.c b/src/fns.c index 6b1f7331f5..8c6894b77d 100644 --- a/src/fns.c +++ b/src/fns.c @@ -3169,7 +3169,7 @@ #define MIME_LINE_LENGTH 76 #define IS_ASCII(Character) \ ((Character) < 128) #define IS_BASE64(Character) \ - (IS_ASCII (Character) && base64_char_to_value[Character] >= 0) + (IS_ASCII (Character) && b64_char_to_value[Character] >= 0) #define IS_BASE64_IGNORABLE(Character) \ ((Character) == ' ' || (Character) == '\t' || (Character) == '\n' \ || (Character) == '\f' || (Character) == '\r') @@ -3202,6 +3202,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ '8', '9', '+', '/' /* 60-63 */ }; +static const char base64url_value_to_char[64] = +{ + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', /* 0- 9 */ + 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', /* 10-19 */ + 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', /* 20-29 */ + 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', /* 30-39 */ + 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', /* 40-49 */ + 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', /* 50-59 */ + '8', '9', '-', '_' /* 60-63 */ +}; + /* Table of base64 values for first 128 characters. */ static const short base64_char_to_value[128] = { @@ -3220,6 +3231,23 @@ #define READ_QUADRUPLET_BYTE(retval) \ 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ }; +static const short base64url_char_to_value[128] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 0- 9 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 10- 19 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 20- 29 */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 30- 39 */ + -1, -1, -1, -1, -1, 62, -1, -1, 52, 53, /* 40- 49 */ + 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, /* 50- 59 */ + -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, /* 60- 69 */ + 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 70- 79 */ + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, /* 80- 89 */ + 25, -1, -1, -1, -1, 63, -1, 26, 27, 28, /* 90- 99 */ + 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, /* 100-109 */ + 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, /* 110-119 */ + 49, 50, 51, -1, -1, -1, -1, -1 /* 120-127 */ +}; + /* The following diagram shows the logical steps by which three octets get transformed into four base64 characters. @@ -3239,9 +3267,17 @@ #define READ_QUADRUPLET_BYTE(retval) \ base64 characters. */ -static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool); +static ptrdiff_t base64_encode_1 (const char *, char *, ptrdiff_t, bool, bool, + bool, bool); static ptrdiff_t base64_decode_1 (const char *, char *, ptrdiff_t, bool, - ptrdiff_t *); + bool, ptrdiff_t *); + +Lisp_Object base64_encode_region_1 (Lisp_Object, Lisp_Object, bool, + bool, bool); + +Lisp_Object base64_encode_string_1(Lisp_Object, bool, + bool, bool); + DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, 2, 3, "r", @@ -3250,6 +3286,26 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, Optional third argument NO-LINE-BREAK means do not break long lines into shorter lines. */) (Lisp_Object beg, Lisp_Object end, Lisp_Object no_line_break) +{ + return base64_encode_region_1(beg, end, NILP (no_line_break), true, false); +} + + +DEFUN ("base64url-encode-region", Fbase64url_encode_region, Sbase64url_encode_region, + 2, 3, "r", + doc: /* Base64url-encode the region between BEG and END. +Return the length of the encoded text. +Optional second argument NO-PAD means do not add padding char =. + +This is the variant defined in RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object no_pad) +{ + return base64_encode_region_1(beg, end, false, NILP(no_pad), true); +} + +Lisp_Object +base64_encode_region_1 (Lisp_Object beg, Lisp_Object end, bool line_break, + bool pad, bool base64url) { char *encoded; ptrdiff_t allength, length; @@ -3272,7 +3328,8 @@ DEFUN ("base64-encode-region", Fbase64_encode_region, Sbase64_encode_region, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 ((char *) BYTE_POS_ADDR (ibeg), - encoded, length, NILP (no_line_break), + encoded, length, line_break, + pad, base64url, !NILP (BVAR (current_buffer, enable_multibyte_characters))); if (encoded_length > allength) emacs_abort (); @@ -3310,6 +3367,26 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, into shorter lines. */) (Lisp_Object string, Lisp_Object no_line_break) { + + return base64_encode_string_1(string, NILP (no_line_break), true, false); +} + +DEFUN ("base64url-encode-string", Fbase64url_encode_string, Sbase64url_encode_string, + 1, 2, 0, + doc: /* Base64url-encode STRING and return the result. +Optional second argument NO-PAD means do not add padding char =. + +This is the variant defined in RFC4648. */) + (Lisp_Object string, Lisp_Object no_pad) +{ + + return base64_encode_string_1(string, false, NILP(no_pad), true); +} + +Lisp_Object +base64_encode_string_1(Lisp_Object string, bool line_break, + bool pad, bool base64url) +{ ptrdiff_t allength, length, encoded_length; char *encoded; Lisp_Object encoded_string; @@ -3328,7 +3405,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, encoded = SAFE_ALLOCA (allength); encoded_length = base64_encode_1 (SSDATA (string), - encoded, length, NILP (no_line_break), + encoded, length, line_break, + pad, base64url, STRING_MULTIBYTE (string)); if (encoded_length > allength) emacs_abort (); @@ -3347,7 +3425,8 @@ DEFUN ("base64-encode-string", Fbase64_encode_string, Sbase64_encode_string, static ptrdiff_t base64_encode_1 (const char *from, char *to, ptrdiff_t length, - bool line_break, bool multibyte) + bool line_break, bool pad, bool base64url, + bool multibyte) { int counter = 0; ptrdiff_t i = 0; @@ -3355,6 +3434,7 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, int c; unsigned int value; int bytes; + char const *b64_value_to_char = (base64url) ? base64url_value_to_char : base64_value_to_char; while (i < length) { @@ -3385,16 +3465,19 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, /* Process first byte of a triplet. */ - *e++ = base64_value_to_char[0x3f & c >> 2]; + *e++ = b64_value_to_char[0x3f & c >> 2]; value = (0x03 & c) << 4; /* Process second byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + *e++ = '='; + } break; } @@ -3410,15 +3493,18 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x0f & c >> 4)]; + *e++ = b64_value_to_char[value | (0x0f & c >> 4)]; value = (0x0f & c) << 2; /* Process third byte of a triplet. */ if (i == length) { - *e++ = base64_value_to_char[value]; - *e++ = '='; + *e++ = b64_value_to_char[value]; + if (pad) + { + *e++ = '='; + } break; } @@ -3434,8 +3520,8 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, else c = from[i++]; - *e++ = base64_value_to_char[value | (0x03 & c >> 6)]; - *e++ = base64_value_to_char[0x3f & c]; + *e++ = b64_value_to_char[value | (0x03 & c >> 6)]; + *e++ = b64_value_to_char[0x3f & c]; } return e - to; @@ -3443,11 +3529,13 @@ base64_encode_1 (const char *from, char *to, ptrdiff_t length, DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, - 2, 2, "r", + 2, 3, "r", doc: /* Base64-decode the region between BEG and END. Return the length of the decoded text. -If the region can't be decoded, signal an error and don't modify the buffer. */) - (Lisp_Object beg, Lisp_Object end) +If the region can't be decoded, signal an error and don't modify the buffer. +Optional third argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object beg, Lisp_Object end, Lisp_Object base64url) { ptrdiff_t ibeg, iend, length, allength; char *decoded; @@ -3472,7 +3560,7 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, move_gap_both (XFIXNAT (beg), ibeg); decoded_length = base64_decode_1 ((char *) BYTE_POS_ADDR (ibeg), - decoded, length, + decoded, length, !NILP (base64url), multibyte, &inserted_chars); if (decoded_length > allength) emacs_abort (); @@ -3506,9 +3594,11 @@ DEFUN ("base64-decode-region", Fbase64_decode_region, Sbase64_decode_region, } DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, - 1, 1, 0, - doc: /* Base64-decode STRING and return the result. */) - (Lisp_Object string) + 1, 2, 0, + doc: /* Base64-decode STRING and return the result +Optional argument BASE64URL define if base64Url variant will be used +see RFC4648. */) + (Lisp_Object string, Lisp_Object base64url) { char *decoded; ptrdiff_t length, decoded_length; @@ -3523,7 +3613,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, /* The decoded result should be unibyte. */ decoded_length = base64_decode_1 (SSDATA (string), decoded, length, - 0, NULL); + !NILP (base64url), 0, NULL); if (decoded_length > length) emacs_abort (); else if (decoded_length >= 0) @@ -3545,6 +3635,7 @@ DEFUN ("base64-decode-string", Fbase64_decode_string, Sbase64_decode_string, static ptrdiff_t base64_decode_1 (const char *from, char *to, ptrdiff_t length, + bool base64url, bool multibyte, ptrdiff_t *nchars_return) { ptrdiff_t i = 0; /* Used inside READ_QUADRUPLET_BYTE */ @@ -3552,6 +3643,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, unsigned char c; unsigned long value; ptrdiff_t nchars = 0; + short const *b64_char_to_value = (base64url) ? base64url_char_to_value : base64_char_to_value; while (1) { @@ -3561,7 +3653,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value = base64_char_to_value[c] << 18; + value = b64_char_to_value[c] << 18; /* Process second byte of a quadruplet. */ @@ -3569,7 +3661,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 12; + value |= b64_char_to_value[c] << 12; c = (unsigned char) (value >> 16); if (multibyte && c >= 128) @@ -3580,7 +3672,14 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process third byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') { @@ -3593,7 +3692,7 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c] << 6; + value |= b64_char_to_value[c] << 6; c = (unsigned char) (0xff & value >> 8); if (multibyte && c >= 128) @@ -3604,14 +3703,21 @@ base64_decode_1 (const char *from, char *to, ptrdiff_t length, /* Process fourth byte of a quadruplet. */ - READ_QUADRUPLET_BYTE (-1); + if (!base64url) + { + READ_QUADRUPLET_BYTE (-1); + } + else + { + READ_QUADRUPLET_BYTE (e-to); + } if (c == '=') continue; if (!IS_BASE64 (c)) return -1; - value |= base64_char_to_value[c]; + value |= b64_char_to_value[c]; c = (unsigned char) (0xff & value); if (multibyte && c >= 128) @@ -5445,6 +5551,8 @@ syms_of_fns (void) defsubr (&Sbase64_decode_region); defsubr (&Sbase64_encode_string); defsubr (&Sbase64_decode_string); + defsubr (&Sbase64url_encode_region); + defsubr (&Sbase64url_encode_string); defsubr (&Smd5); defsubr (&Ssecure_hash_algorithms); defsubr (&Ssecure_hash); diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el index 6ebab4287f..dab43212f5 100644 --- a/test/src/fns-tests.el +++ b/test/src/fns-tests.el @@ -233,6 +233,185 @@ fns-tests-func-arity (should (equal (func-arity (eval (lambda (x &optional y)) t)) '(1 . 2))) (should (equal (func-arity 'let) '(1 . unevalled)))) +(defun fns-tests--string-repeat (s o) + (apply 'concat (make-list o s))) + +(defmacro fns-tests--with-region (funcname string &rest args) + "Apply FUNCNAME in a temp bufer on the region produced by STRING." + (declare (indent 1)) + `(with-temp-buffer + (insert ,string) + (,funcname (point-min) (point-max) ,@args) + (buffer-string))) + +(ert-deftest fns-tests-base64-encode-region () + ;; standard variant RFC2045 + (should (equal (fns-tests--with-region base64-encode-region "") "")) + (should (equal (fns-tests--with-region base64-encode-region "f") "Zg==")) + (should (equal (fns-tests--with-region base64-encode-region "fo") "Zm8=")) + (should (equal (fns-tests--with-region base64-encode-region "foo") "Zm9v")) + (should (equal (fns-tests--with-region base64-encode-region "foob") "Zm9vYg==")) + (should (equal (fns-tests--with-region base64-encode-region "fooba") "Zm9vYmE=")) + (should (equal (fns-tests--with-region base64-encode-region "foobar") "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64-encode-region "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (fns-tests--with-region base64-encode-region "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/"))) + +(ert-deftest fns-tests-base64-encode-string () + ;; standard variant RFC2045 + (should (equal (base64-encode-string "") "")) + (should (equal (base64-encode-string "f") "Zg==")) + (should (equal (base64-encode-string "fo") "Zm8=")) + (should (equal (base64-encode-string "foo") "Zm9v")) + (should (equal (base64-encode-string "foob") "Zm9vYg==")) + (should (equal (base64-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l+")) + (should (equal (base64-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l/"))) + +(ert-deftest fns-test-base64url-encode-region () + ;; url variant wih padding + (should (equal (fns-tests--with-region base64url-encode-region "") "")) + (should (equal (fns-tests--with-region base64url-encode-region "f") "Zg==")) + (should (equal (fns-tests--with-region base64url-encode-region "fo") "Zm8=")) + (should (equal (fns-tests--with-region base64url-encode-region "foo") "Zm9v")) + (should (equal (fns-tests--with-region base64url-encode-region "foob") "Zm9vYg==")) + (should (equal (fns-tests--with-region base64url-encode-region "fooba") "Zm9vYmE=")) + (should (equal (fns-tests--with-region base64url-encode-region "foobar") "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l-")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l_")) + + ;; url variant no padding + (should (equal (fns-tests--with-region base64url-encode-region "" t) "")) + (should (equal (fns-tests--with-region base64url-encode-region "f" t) "Zg")) + (should (equal (fns-tests--with-region base64url-encode-region "fo" t) "Zm8")) + (should (equal (fns-tests--with-region base64url-encode-region "foo" t) "Zm9v")) + (should (equal (fns-tests--with-region base64url-encode-region "foob" t) "Zm9vYg")) + (should (equal (fns-tests--with-region base64url-encode-region "fooba" t) "Zm9vYmE")) + (should (equal (fns-tests--with-region base64url-encode-region "foobar" t) "Zm9vYmFy")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7e" t) "FPucA9l-")) + (should (equal (fns-tests--with-region base64url-encode-region "\x14\xfb\x9c\x03\xd9\x7f" t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "f" 100) t) + (concat (fns-tests--string-repeat "Zm" 66) "Zg"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "fo" 50) t) + (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foo" 25) t) + (fns-tests--string-repeat "Zm9v" 25))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foob" 15) t) + (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "fooba" 15) t) + (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "foobar" 15) t) + (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) + (fns-tests--string-repeat "FPucA9l-" 10))) + (should (equal (fns-tests--with-region base64url-encode-region (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) + (fns-tests--string-repeat "FPucA9l_" 10)))) + +(ert-deftest fns-test-base64url-encode-string () + ;; url variant wih padding + (should (equal (base64url-encode-string "") "")) + (should (equal (base64url-encode-string "f") "Zg==")) + (should (equal (base64url-encode-string "fo") "Zm8=")) + (should (equal (base64url-encode-string "foo") "Zm9v")) + (should (equal (base64url-encode-string "foob") "Zm9vYg==")) + (should (equal (base64url-encode-string "fooba") "Zm9vYmE=")) + (should (equal (base64url-encode-string "foobar") "Zm9vYmFy")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7e") "FPucA9l-")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7f") "FPucA9l_")) + + ;; url variant no padding + (should (equal (base64url-encode-string "" t) "")) + (should (equal (base64url-encode-string "f" t) "Zg")) + (should (equal (base64url-encode-string "fo" t) "Zm8")) + (should (equal (base64url-encode-string "foo" t) "Zm9v")) + (should (equal (base64url-encode-string "foob" t) "Zm9vYg")) + (should (equal (base64url-encode-string "fooba" t) "Zm9vYmE")) + (should (equal (base64url-encode-string "foobar" t) "Zm9vYmFy")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7e" t) "FPucA9l-")) + (should (equal (base64url-encode-string "\x14\xfb\x9c\x03\xd9\x7f" t) "FPucA9l_")) + + + ;; url variant no line break no padding + (should (equal (base64url-encode-string (fns-tests--string-repeat "f" 100) t) (concat (fns-tests--string-repeat "Zm" 66) "Zg"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "fo" 50) t) (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foo" 25) t) (fns-tests--string-repeat "Zm9v" 25))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foob" 15) t) (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "fooba" 15) t) (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "foobar" 15) t) (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy"))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10) t) (fns-tests--string-repeat "FPucA9l-" 10))) + (should (equal (base64url-encode-string (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10) t) (fns-tests--string-repeat "FPucA9l_" 10)))) + +(ert-deftest fns-tests-base64-decode-string () + ;; standard variant RFC2045 + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==") "f")) + (should (equal (base64-decode-string "Zm8=") "fo")) + (should (equal (base64-decode-string "Zm9v") "foo")) + (should (equal (base64-decode-string "Zm9vYg==") "foob")) + (should (equal (base64-decode-string "Zm9vYmE=") "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy") "foobar")) + (should (equal (base64-decode-string "FPucA9l+") "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l/") "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; no paddign + (should (equal (base64-decode-string "" t) "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + + ;; url variant wih padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg==" t) "f") ) + (should (equal (base64-decode-string "Zm8=" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg==" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE=" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + ;; url variant no padding + (should (equal (base64-decode-string "") "")) + (should (equal (base64-decode-string "Zg" t) "f")) + (should (equal (base64-decode-string "Zm8" t) "fo")) + (should (equal (base64-decode-string "Zm9v" t) "foo")) + (should (equal (base64-decode-string "Zm9vYg" t) "foob")) + (should (equal (base64-decode-string "Zm9vYmE" t) "fooba")) + (should (equal (base64-decode-string "Zm9vYmFy" t) "foobar")) + (should (equal (base64-decode-string "FPucA9l-" t) "\x14\xfb\x9c\x03\xd9\x7e")) + (should (equal (base64-decode-string "FPucA9l_" t) "\x14\xfb\x9c\x03\xd9\x7f")) + + + ;; url variant no line break no padding + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm" 66) "Zg") t) + (fns-tests--string-repeat "f" 100))) + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm9mb2Zv" 16) "Zm9mbw") t) + (fns-tests--string-repeat "fo" 50))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9v" 25) t) + (fns-tests--string-repeat "foo" 25))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9vYmZvb2Jmb29i" 5) t) + (fns-tests--string-repeat "foob" 15))) + (should (equal (base64-decode-string (fns-tests--string-repeat "Zm9vYmFmb29iYWZvb2Jh" 5) t) + (fns-tests--string-repeat "fooba" 15))) + (should (equal (base64-decode-string (concat (fns-tests--string-repeat "Zm9vYmFyZm9vYmFy" 7) "Zm9vYmFy") t) + (fns-tests--string-repeat "foobar" 15))) + (should (equal (base64-decode-string (fns-tests--string-repeat "FPucA9l-" 10) t) + (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7e" 10))) + (should (equal (base64-decode-string (fns-tests--string-repeat "FPucA9l_" 10) t) + (fns-tests--string-repeat "\x14\xfb\x9c\x03\xd9\x7f" 10))) + + ;; errors check + (should (eq :got-error (condition-case () (base64-decode-string "Zg=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmE") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zm9vYmFy=") (error :got-error)))) + (should (eq :got-error (condition-case () (base64-decode-string "Zg=Zg=") (error :got-error))))) + (ert-deftest fns-tests-hash-buffer () (should (equal (sha1 "foo") "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33")) (should (equal (with-temp-buffer -- 2.21.0 [-- Attachment #5: Type: text/plain, Size: 42 bytes --] Tell me what you thing is the way to go. ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-27 20:30 ` Pierre Téchoueyres @ 2019-06-07 21:04 ` Pierre Téchoueyres 2019-06-08 5:52 ` Eli Zaretskii 2019-06-08 8:18 ` Eli Zaretskii 1 sibling, 1 reply; 23+ messages in thread From: Pierre Téchoueyres @ 2019-06-07 21:04 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel Had you have a chance to look at my patches ? Should/Could I do something ? pierre.techoueyres@free.fr (Pierre Téchoueyres) writes: > Eli Zaretskii <eliz@gnu.org> writes: > >>> From: pierre.techoueyres@free.fr (Pierre Téchoueyres) >>> Cc: emacs-devel@gnu.org >>> Date: Thu, 23 May 2019 21:37:01 +0200 >>> >>> > I'd suggest to call the new argument base64url or somesuch, since >>> > this is trhe official name. >>> > >>> You mean in replacement of url_variant or b64_value_to_char ? >> >> The former. >> >>> How should I send new versions of the patch ? As a full patch in >>> attachment like previously ? >> >> Yes. And please include the log messages (in the ChangeLog style). >> >> > You'll find two patches attached. > First one contains requested changes and, hoppefully, a valid Changelog > > > > The second patch contains an reworked version wich instead of adding > many parameters to base64-encode-region (resp. base64-encode-string) > function create the base64url-encode-region > (resp. base64url-encode-string) function. Documentation and tests are > also updated consequently. > > > > Tell me what you thing is the way to go. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-07 21:04 ` Pierre Téchoueyres @ 2019-06-08 5:52 ` Eli Zaretskii 0 siblings, 0 replies; 23+ messages in thread From: Eli Zaretskii @ 2019-06-08 5:52 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: pierre.techoueyres@free.fr (Pierre Téchoueyres) > Cc: emacs-devel@gnu.org > Date: Fri, 07 Jun 2019 23:04:44 +0200 > > Had you have a chance to look at my patches ? > Should/Could I do something ? Your patches are in my queue (which was unusually long the past 2 weeks). Sorry for the delay, I will get to that in a day or two. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-05-27 20:30 ` Pierre Téchoueyres 2019-06-07 21:04 ` Pierre Téchoueyres @ 2019-06-08 8:18 ` Eli Zaretskii 2019-06-11 18:36 ` Pierre Téchoueyres 1 sibling, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-06-08 8:18 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: emacs-devel > From: pierre.techoueyres@free.fr (Pierre Téchoueyres) > Cc: emacs-devel@gnu.org > Date: Mon, 27 May 2019 22:30:14 +0200 > > You'll find two patches attached. Thanks, I liked the second one better, so I pushed it. Please note that the doc strings and the manual text included a few typos, and also used the passive tense too much. See the follow-up changes I pushed after your changeset. In the future, I suggest to use the Emacs spell-checking capabilities to spell-check the text in the manual and also the comments/strings in the C/Lisp sources. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-08 8:18 ` Eli Zaretskii @ 2019-06-11 18:36 ` Pierre Téchoueyres 2019-06-11 18:42 ` Eli Zaretskii 0 siblings, 1 reply; 23+ messages in thread From: Pierre Téchoueyres @ 2019-06-11 18:36 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Paul Eggert, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> From: pierre.techoueyres@free.fr (Pierre Téchoueyres) >> Cc: emacs-devel@gnu.org >> Date: Mon, 27 May 2019 22:30:14 +0200 >> >> You'll find two patches attached. > > Thanks, I liked the second one better, so I pushed it. Thank you. Honestly I didn't expect it was accepted without further modifications. > > Please note that the doc strings and the manual text included a few > typos, and also used the passive tense too much. See the follow-up > changes I pushed after your changeset. In the future, I suggest to > use the Emacs spell-checking capabilities to spell-check the text in > the manual and also the comments/strings in the C/Lisp sources. This was the part I was the less confident (and for good reasons). Thanks again to have (completely) rewritten it. I'll try to be more rigorous in the future with the doc, NEWS, etc.. I'll also thanks Paul Eggert for his optimizations of the new functions in commit 5abaea334cf4c0e004fca2b8b272e091eb5b5444. Pierre. P.S. Would it be any interest if I try to add others bases encoding as defined in RFC 4648 ? I think, first, to BASE32 and maybe on BASE16 later. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-11 18:36 ` Pierre Téchoueyres @ 2019-06-11 18:42 ` Eli Zaretskii 2019-06-11 18:47 ` Achim Gratz 0 siblings, 1 reply; 23+ messages in thread From: Eli Zaretskii @ 2019-06-11 18:42 UTC (permalink / raw) To: Pierre Téchoueyres; +Cc: eggert, emacs-devel > From: pierre.techoueyres@free.fr (Pierre Téchoueyres) > Cc: emacs-devel@gnu.org, Paul Eggert <eggert@Penguin.CS.UCLA.EDU> > Date: Tue, 11 Jun 2019 20:36:52 +0200 > > Would it be any interest if I try to add others bases encoding as > defined in RFC 4648 ? I think, first, to BASE32 and maybe on BASE16 > later. I have never needed those. Let's hear what others have to say. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-11 18:42 ` Eli Zaretskii @ 2019-06-11 18:47 ` Achim Gratz 2019-06-11 20:14 ` Richard Copley 2019-06-12 6:50 ` Stefan Monnier 0 siblings, 2 replies; 23+ messages in thread From: Achim Gratz @ 2019-06-11 18:47 UTC (permalink / raw) To: emacs-devel Eli Zaretskii writes: >> From: pierre.techoueyres@free.fr (Pierre Téchoueyres) >> Cc: emacs-devel@gnu.org, Paul Eggert <eggert@Penguin.CS.UCLA.EDU> >> Date: Tue, 11 Jun 2019 20:36:52 +0200 >> >> Would it be any interest if I try to add others bases encoding as >> defined in RFC 4648 ? I think, first, to BASE32 and maybe on BASE16 >> later. > > I have never needed those. Let's hear what others have to say. Base32url is useful for use on case-insensitive filesystems or with case-insensitive protocols. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-11 18:47 ` Achim Gratz @ 2019-06-11 20:14 ` Richard Copley 2019-06-12 15:34 ` Eli Zaretskii 2019-06-12 6:50 ` Stefan Monnier 1 sibling, 1 reply; 23+ messages in thread From: Richard Copley @ 2019-06-11 20:14 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Emacs Development [-- Attachment #1: Type: text/plain, Size: 1097 bytes --] Eli Zaretskii writes: Please note that the doc strings and the manual text included a few typos, and also used the passive tense too much. See the follow-up changes I pushed after your changeset. In the future, I suggest to use the Emacs spell-checking capabilities to spell-check the text in the manual and also the comments/strings in the C/Lisp sources. Two more typos (if/of, inserts/insert): diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi index 55f5269849..2e7c497f57 100644 --- a/doc/lispref/text.texi +++ b/doc/lispref/text.texi @@ -4560,8 +4560,8 @@ Base 64 @deffn Command base64url-encode-region beg end &optional no-pad This function is like @code{base64-encode-region}, but it implements -the URL variant if base 64 encoding, per RFC 4648, and it doesn't -inserts newline characters into the encoded text, so the output is +the URL variant of base 64 encoding, per RFC 4648, and it doesn't +insert newline characters into the encoded text, so the output is just one long line. If the optional argument @var{no-pad} is non-@code{nil} then this [-- Attachment #2: Type: text/html, Size: 1249 bytes --] ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-11 20:14 ` Richard Copley @ 2019-06-12 15:34 ` Eli Zaretskii 0 siblings, 0 replies; 23+ messages in thread From: Eli Zaretskii @ 2019-06-12 15:34 UTC (permalink / raw) To: Richard Copley; +Cc: emacs-devel > From: Richard Copley <rcopley@gmail.com> > Date: Tue, 11 Jun 2019 21:14:11 +0100 > Cc: Emacs Development <emacs-devel@gnu.org> > > Two more typos (if/of, inserts/insert): Thanks, fixed. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-11 18:47 ` Achim Gratz 2019-06-11 20:14 ` Richard Copley @ 2019-06-12 6:50 ` Stefan Monnier 2019-06-12 19:24 ` Achim Gratz 1 sibling, 1 reply; 23+ messages in thread From: Stefan Monnier @ 2019-06-12 6:50 UTC (permalink / raw) To: emacs-devel >>> Would it be any interest if I try to add others bases encoding as >>> defined in RFC 4648 ? I think, first, to BASE32 and maybe on BASE16 >>> later. >> I have never needed those. Let's hear what others have to say. Same here. > Base32url is useful for use on case-insensitive filesystems or > with case-insensitive protocols. That's what it's designed for, yes, but I can't think of any example where that's useful. E.g. for case-insensitive filenames, using a hash instead of a base32 encoding was a better option in all the cases I've encountered because I only needed "one-way" and it conveniently bounds the max file name length. Stefan ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-12 6:50 ` Stefan Monnier @ 2019-06-12 19:24 ` Achim Gratz 2019-06-12 21:50 ` Stefan Monnier 0 siblings, 1 reply; 23+ messages in thread From: Achim Gratz @ 2019-06-12 19:24 UTC (permalink / raw) To: emacs-devel Stefan Monnier writes: >> Base32url is useful for use on case-insensitive filesystems or >> with case-insensitive protocols. > > That's what it's designed for, yes, but I can't think of any example > where that's useful. E.g. for case-insensitive filenames, using a hash > instead of a base32 encoding was a better option in all the cases I've > encountered because I only needed "one-way" and it conveniently bounds > the max file name length. I've used it for sharding into a blobstore on NTFS/SMB (two characters for first level, so tree width of 1024). The key is indeed a hash (SHA512), so the probability of a collision would be low enough to just ignore the case (for only 512 wide tree), but it felt cleaner to encode via base32url instead. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Add support for base64url variant 2019-06-12 19:24 ` Achim Gratz @ 2019-06-12 21:50 ` Stefan Monnier 0 siblings, 0 replies; 23+ messages in thread From: Stefan Monnier @ 2019-06-12 21:50 UTC (permalink / raw) To: emacs-devel > I've used it for sharding into a blobstore on NTFS/SMB (two characters > for first level, so tree width of 1024). The key is indeed a hash > (SHA512), so the probability of a collision would be low enough to just > ignore the case (for only 512 wide tree), but it felt cleaner to encode > via base32url instead. Ah, I was assuming a hex representation of the hash. In that case, using base32 is basically an optimization that shortens the filename by ~20% [ Using base64 would make the hash a bit weaker and shorten the filename by ~33% instead. ] Stefan ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2019-06-12 21:50 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-21 22:32 Add support for base64url variant Pierre Téchoueyres 2019-05-22 7:43 ` Eli Zaretskii 2019-05-22 9:25 ` Pierre Téchoueyres 2019-05-22 9:50 ` Eli Zaretskii 2019-05-23 17:51 ` Pierre Téchoueyres 2019-05-23 18:45 ` Noam Postavsky 2019-05-23 19:32 ` Pierre Téchoueyres 2019-05-23 18:58 ` Eli Zaretskii 2019-05-23 18:50 ` Eli Zaretskii 2019-05-23 19:37 ` Pierre Téchoueyres 2019-05-23 19:51 ` Eli Zaretskii 2019-05-27 20:30 ` Pierre Téchoueyres 2019-06-07 21:04 ` Pierre Téchoueyres 2019-06-08 5:52 ` Eli Zaretskii 2019-06-08 8:18 ` Eli Zaretskii 2019-06-11 18:36 ` Pierre Téchoueyres 2019-06-11 18:42 ` Eli Zaretskii 2019-06-11 18:47 ` Achim Gratz 2019-06-11 20:14 ` Richard Copley 2019-06-12 15:34 ` Eli Zaretskii 2019-06-12 6:50 ` Stefan Monnier 2019-06-12 19:24 ` Achim Gratz 2019-06-12 21:50 ` Stefan Monnier
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).