From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Philip Kaludercic Newsgroups: gmane.emacs.bugs Subject: bug#70312: [PATCH v2] Avoid unnecessary escaping in url-build-query-string Date: Sun, 19 May 2024 11:18:47 +0000 Message-ID: <87bk52uk14.fsf@posteo.net> References: <8734ruk3tf.fsf@wibble.ilmari.org> <87zfu2ij4s.fsf@wibble.ilmari.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27156"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 70312@debbugs.gnu.org To: Dagfinn Ilmari =?UTF-8?Q?Manns=C3=A5ker?= Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun May 19 13:19:14 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1s8eZN-0006mc-NY for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 19 May 2024 13:19:13 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s8eZA-0006LZ-V4; Sun, 19 May 2024 07:19:00 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s8eZ8-0006GR-3k for bug-gnu-emacs@gnu.org; Sun, 19 May 2024 07:18:58 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s8eZ7-0005XS-Rl for bug-gnu-emacs@gnu.org; Sun, 19 May 2024 07:18:57 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s8eZB-0004PJ-Pl for bug-gnu-emacs@gnu.org; Sun, 19 May 2024 07:19:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Philip Kaludercic Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 19 May 2024 11:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70312 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 70312-submit@debbugs.gnu.org id=B70312.171611754116935 (code B ref 70312); Sun, 19 May 2024 11:19:01 +0000 Original-Received: (at 70312) by debbugs.gnu.org; 19 May 2024 11:19:01 +0000 Original-Received: from localhost ([127.0.0.1]:35968 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s8eZA-0004P5-N6 for submit@debbugs.gnu.org; Sun, 19 May 2024 07:19:01 -0400 Original-Received: from mout01.posteo.de ([185.67.36.65]:53889) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s8eZ8-0004Oz-5e for 70312@debbugs.gnu.org; Sun, 19 May 2024 07:18:59 -0400 Original-Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id F094F240027 for <70312@debbugs.gnu.org>; Sun, 19 May 2024 13:18:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1716117528; bh=tWiNVzxFT9sX0uFlQbENM9HIJIiXomFzE55YS2tPp+s=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=PMqOcm4GXSS7v4QDz7XAXlcLZ65kUrmlMh/lcCGyAcsXebBYTVk/PeGIVZVwXtsjT bKz0ZsfbcUMaWR/JZjxganckPhii0fNTYv+lYAWnAj9UGW56E6XSEBylzNUa4atHDY HNf1eXgfcUO7zZTc1/kZ+ZRYZK3enyno/u7lLqohqpE6ms/kh7jD7zVJeObk1p8HRD kphXdQ9YVb0SGyDPF2cRqqBBq3+5COmx61A9nhhXrslsMIja+jAcAdvJ6qU9E/qrhw sWH8QSddRlL4MDc+sp+D6uGSpbCGpgcxQSb/DoiN+I+XSZqCMiA4CPlvJg3YHdzhiX wbjxdS3QjVMAg== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Vhyr73kZFz9rxD; Sun, 19 May 2024 13:18:47 +0200 (CEST) In-Reply-To: <87zfu2ij4s.fsf@wibble.ilmari.org> ("Dagfinn Ilmari =?UTF-8?Q?Manns=C3=A5ker?="'s message of "Tue, 09 Apr 2024 17:41:07 +0100") X-Hashcash: 1:20:240519:70312@debbugs.gnu.org::DYlLrWY9wnUHdTvq:9E9 X-Hashcash: 1:20:240519:ilmari@ilmari.org::FlM6eWcHE+IntaYr:3Nh2 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:285423 Archived-At: Dagfinn Ilmari Manns=C3=A5ker writes: > Hi again, > > I realised I'd forgotten to add tests, and that made me realise that > url-query-allowed-chars is not correct for this, since that also > contains '=3D', '&', and ';'. So here's an updated patch, which creates a > new url-query-key-value-allowed-chars constant, which is > url-query-allowed-chars minus the aforementioned three chars, and adds > tests covering that, for both keys and values. This patch breaks a script I have that authenticates via HTTP. Apparently it doesn't escape enough now: (url-build-query-string '((var "\"$%&')+:;<>?@]^{|}"))) "var=3D%22$%%26')+:%3B%3C%3E?@%5D%5E%7B%7C%7D" whereas it used to be: (url-build-query-string '((var "\"$%&')+:;<>?@]^{|}"))) "var=3D%22%24%25%26%27%29%2B%3A%3B%3C%3E%3F%40%5D%5E%7B%7C%7D" If it is true, that it just unnecessarily escapes too much (and this is not a problem), then I'd suggest reverting the patch as the easiest solution to avoid breakage in the long term. > > - ilmari > >>>From 89db0a1226d8d7cca1846e9c737d4a67c971ec75 Mon Sep 17 00:00:00 2001 > From: =3D?UTF-8?q?Dagfinn=3D20Ilmari=3D20Manns=3DC3=3DA5ker?=3D > Date: Tue, 9 Apr 2024 15:02:45 +0100 > Subject: [PATCH v2] Avoid unnecessary escaping in url-build-query-string > > * lisp/url/url-util.el (url-build-query-string): > Create a new url-query-key-value-allowed-chars constant and pass that to > url-hexify-string to avoid unnecessarily escaping characters that don't > need to be escaped in query string keys and values. > * test/lisp/url/url-util-tests.el (url-util-tests): > Add test cases. > --- > lisp/url/url-util.el | 12 +++++++++++- > test/lisp/url/url-util-tests.el | 6 +++++- > 2 files changed, 16 insertions(+), 2 deletions(-) > > diff --git a/lisp/url/url-util.el b/lisp/url/url-util.el > index 5f45b98c7a5..f063efe18a6 100644 > --- a/lisp/url/url-util.el > +++ b/lisp/url/url-util.el > @@ -268,7 +268,8 @@ url-build-query-string > (lambda (key-vals) > (let ((escaped > (mapcar (lambda (sym) > - (url-hexify-string (format "%s" sym))) key-vals))) > + (url-hexify-string (format "%s" sym) url-query-key= -value-allowed-chars)) > + key-vals))) > (mapconcat (lambda (val) > (let ((vprint (format "%s" val)) > (eprint (format "%s" (car escaped)))) > @@ -410,6 +411,15 @@ url-query-allowed-chars > "Allowed-character byte mask for the query segment of a URI. > These characters are specified in RFC 3986, Appendix A.") >=20=20 > +(defconst url-query-key-value-allowed-chars > + (let ((vec (copy-sequence url-query-allowed-chars))) > + (aset vec ?=3D nil) > + (aset vec ?& nil) > + (aset vec ?\; nil) > + vec) > + "Allowed-charcter byte mask for keys and values in the query segment o= f a URI. > +url-query-allowed-chars minus '=3D', '&', and ';'.") > + > ;;;###autoload > (defun url-encode-url (url) > "Return a properly URI-encoded version of URL. > diff --git a/test/lisp/url/url-util-tests.el b/test/lisp/url/url-util-tes= ts.el > index 133aa0ffd88..c6246d69a2a 100644 > --- a/test/lisp/url/url-util-tests.el > +++ b/test/lisp/url/url-util-tests.el > @@ -32,7 +32,11 @@ url-util-tests > ("key1=3Dval1;key2=3Dval2;key3=3Dval1;key3=3Dval2;key4;key5" > ((key1 "val1") (key2 val2) (key3 val1 val2) ("key4") (key5 "= ")) t) > ("key1=3Dval1;key2=3Dval2;key3=3Dval1;key3=3Dval2;key4=3D;key= 5=3D" > - ((key1 val1) (key2 val2) ("key3" val1 val2) (key4) (key5 "")= ) t t))) > + ((key1 val1) (key2 val2) ("key3" val1 val2) (key4) (key5 "")= ) t t) > + ("key1=3Dval/slash;key2=3Dval%3Bsemi;key3=3Dval%26amp;key4=3D= val%3Deq" > + ((key1 "val/slash") (key2 "val;semi") (key3 "val&") (key4= "val=3Deq")) t) > + ("key%3Deq=3Dval1;key%3Bsemi=3Dval2;key%26amp=3Dval3" > + (("key=3Deq" val1) ("key;semi" val2) ("key&" val3)) t))) > test) > (while tests > (setq test (car tests) --=20 Philip Kaludercic on icterid