From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Daniel Brooks Newsgroups: gmane.emacs.devel Subject: Re: Unicode confusables and reordering characters considered harmful, a simple solution Date: Thu, 04 Nov 2021 19:23:08 -0700 Message-ID: <87k0hnqr1v.fsf@db48x.net> References: <83wnlqk3rn.fsf@gnu.org> <72dd5c2a-42c7-b12e-05ed-e93adbd89727@gmail.com> <83ilxajyhw.fsf@gnu.org> <83fssejxf8.fsf@gnu.org> <835ytajsv2.fsf@gnu.org> <831r3yjqo9.fsf@gnu.org> <83v91aibe7.fsf@gnu.org> <87o872s0wf.fsf_-_@db48x.net> <83lf25gm1j.fsf@gnu.org> <83ee7xgio2.fsf@gnu.org> <87fssdrp54.fsf@db48x.net> <831r3xgfz3.fsf@gnu.org> <87v918qx37.fsf@db48x.net> <83o870fjqg.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="17133"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org, stefan@marxist.se, monnier@iro.umontreal.ca, yuri.v.khan@gmail.com To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Nov 05 03:25:19 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mious-0004CJ-LH for ged-emacs-devel@m.gmane-mx.org; Fri, 05 Nov 2021 03:25:19 +0100 Original-Received: from localhost ([::1]:37354 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1miouq-0007Ma-LA for ged-emacs-devel@m.gmane-mx.org; Thu, 04 Nov 2021 22:25:16 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40188) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1miosv-0006c1-6b for emacs-devel@gnu.org; Thu, 04 Nov 2021 22:23:17 -0400 Original-Received: from smtp-out-4.mxes.net ([2605:d100:2f:10::315]:41002) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mioss-0005Lq-6j for emacs-devel@gnu.org; Thu, 04 Nov 2021 22:23:16 -0400 Original-Received: from Customer-MUA (mua.mxes.net [10.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 4HlknT4GsRz3cCC; Thu, 4 Nov 2021 22:23:09 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mxes.net; s=mta; t=1636078991; bh=cmXUdIRGEqFLRCdc27FAsnRCcq9cU54KIHLey07gQdQ=; h=From:To:Subject:References:Date:In-Reply-To:Message-ID: MIME-Version:Content-Type; b=P8FGwJQ/897eBfT5ca9ATnfYCLWRu0kjrc+P+Sx6tLOElz7WUBgo+aRdmyhlwVl+C vRdfQjkC+wHjhMRWsjOsBKR58IAI6vSrgV7sVqCHfz4DEhijLhWhbp+/nd+Sl8UY7Z zAEOcrQMv/qzWh+nLcTIHGsWrbsOw4A3v0mirrSY= Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGOfPtRkwAAABJQ TFRFpKfbdou67PD6JjJgAwUWXGSeIcyLHgAAAkZJREFUOI1VU8Fy6yAMxLi+Q13fCZ3cnQL3dqTc 7RD+/1feStDXVnXHDuvVSivZTMba2GPdw3gyCGcMAFxTyrTd9dwGoxHiZX9PmRFUHYAQlGGtXY+F Uk0SJOxgJiUEnH1qkitT9D+pQub7qGAmUbR6bu3CvI96Yv6QqkBBMrsyfZccr1/RDXGDTLf4P7ZY glVxe2V+/ACXWO1gvDO9/gDRpFFVmPluvLcmBjd5H6d8DEte+Pbk4rcY/Fa5tLKLOtCZsuQKYhpa LOkYDT7hESya7/WIET3lfQBqX0pwFtbI832Is0ayMUR9B+12xjgPCQ089cfwkCkX6L5TPmRelJTh zMS0Sz1PyjLAMCUWjcmgQLWQMds+e3aaauZDf9dU9A2/8kPVF2odCUoMKHkfjJR+mbgC+DRiycw5 3XSqGe6HmhN/AWjHypkAXOAFW5EiuA1ge2GiZuMb0s1fSEXcATeLUfbyEY2L8yPOmdSsdghQXx3K pz2eoeXuYvMCINVFDrCdNfVUp4eJ6cSEbjbgFjBEvonGGTrgv9cHjAc8aVgSAPoxaONbzfwhDIhR at7IIS7fAGiDSwIA9alhhTBzfA7YM2FY6eMwayrIGK8FDFmshmUA43WqhFtpvoqG9HHaJ7fqtgTz 8EWVkgZgtsylFliHDgk0MB7KAEC45C/rgnGvanNLXyzOeTzcT2nw/N44gfrtYXRQLoz9Q3TgmJRx 2Mx/Q51qzpm+l3m8z2SWBqC5+PZXAtNYlGFf/gKfHfjFkDT4x7od7R+w3Ls+ZdQBuQAAAABJRU5E rkJggg== In-Reply-To: <83o870fjqg.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 04 Nov 2021 09:44:23 +0200") X-Sent-To: Received-SPF: none client-ip=2605:d100:2f:10::315; envelope-from=db48x@db48x.net; helo=smtp-out-4.mxes.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_PASS=-0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:278726 Archived-At: Eli Zaretskii writes: >> From: Daniel Brooks >> Of course it will show them in the comments and strings. > > Then this visual noise will get in the way of people's reading those > comments and strings, and, for strings, will make it very hard to > understand what will be presented to the user when those strings are > output in some UI. > >> That=E2=80=99s where the problem is. > > No, the problem is elsewhere entirely: it's in the punctuation > characters unrelated to strings and comments whose directionality is > overridden, and which thus display in places that cause incorrect > visual interpretation of the program during a casual read. Look at the examples again. In many of them, all of the bidi override characters are inside a string or comment. When that is the case, these characters are only a problem if they cause characters that are inside the string or comment to appear to be outside of it, by reordering those characters relative to the syntactic markers for the string or comment. In other examples these characters are _outside_ the string or comment. Unless Emacs has specific knowledge of the language syntax, showing the characters is the only sure way to know if there is a problem or not. > You misunderstand the cause. The mere presence of these characters is > NOT the root cause. These characters are legitimate and helpful when > used as intended. See TUTORIAL.he for a pertinent example. Please don=E2=80=99t presume to tell me what I do or don=E2=80=99t understa= nd. Yes, there are use cases which are not harmful, but as I have said it must be up to either the programmer or the compiler to answer that question. Emacs doesn=E2=80=99t know the syntax of every programming langua= ge. >> Furthermore, I have not suggested that showing the characters needs to >> preclude any other form of highlighting. If you wish to develop some >> additional way of warning the developer, please do so. > > We are talking about what should be in Emacs. What you suggest > shouldn't. No other suggested feature will be useful to me. This one will. I suggest to you that you do not know what all users want. >> However, I suspect that the compilers for most languages currently in >> active development will develop their own warnings and error messages as >> well. We have plenty of ways for those messages to show up inside Emacs >> as highlights. > > That's a tangent. We are discussing what Emacs should do as a > programmer's editor to flag such suspicious code. That shouldn't need > a compiler if we can do the job ourselves. And we can. This is not a tangent. Emacs relies heavily on compilers and language runtimes for many of its features. This is just one more area where Emacs should not try to be too clever. > >> Rust, for example, has already done so. Here=E2=80=99s an example: >>=20 >> error: unicode codepoint changing visible direction of text present = in comment >> --> src/pathmap/path.rs:10:5 >> | >> 10 | /* } if is_admin begin admins only */ >> | ^^-^^-^^^^^^^^^^--^^^^^^^^^^^^^^^^^^^^ >> | | | | || >> | | | | |'\u{2066}' >> | | | | '\u{2069}' >> | | | '\u{2066}' >> | | '\u{202e}' >> | this comment contains invisible unicode text flow control c= odepoints >> | >> =3D note: `#[deny(text_direction_codepoint_in_comment)]` on by de= fault >> =3D note: these kind of unicode codepoints change the way text >> flows on applications that support them, but can cause confusion >> because they change the order of characters on the screen >> =3D help: if their presence wasn't intentional, you can remove th= em > > Since the Rust compiler evidently does this when it finds these > characters inside comments (and probably also inside strings), IMNSHO > this is a terrible misfeature, because it means code that uses those > controls in legitimate ways cannot be compiled without tweaking > non-default options. That's a cop-out, not the way to flag the > problematic cases. Your conclusion here is incorrect. Rust has choosen a fast strategy, where they implement a broad error today (well, four days ago) knowing that it does not prevent them from introducing a more refined error or set of errors later. Rust also has a very flexible annotation system that allows the programmer to annotate specific statements and language items. If a use of these characters is determined to be legitimate, the programmer can annotate the comment, or the function the comment is in, so that this error is disabled. In projects with strong review culture, seeing that annotation while doing a code review will be a very strong signal that something unusual is going on, and that it needs to be considered carefully. Annotations are are a great feature of Rust that I do not expect Emacs to take into account. Instead I think that Emacs should adopt a similar fast strategy. Anything we do today can be refined later. >> Naturally that already shows up inside of Emacs just fine; see the >> attached image. > > I think this is terrible. At best, it only tells you that something > non-trivial goes on here (but what exactly?). At worst, it looks like > corruption of the source. And while in the malicious case treating > that as corruption is not such a bad idea, all the valid uses of these > characters will also look like corruption. Which means the cure is > probably worse than the disease, because the malicious cases are a > tiny fraction of the valid ones. I cannot believe that you really think this. It shows up with exactly the same highlighting that your recently=E2=80=93introduced highlight-confusing-reorderings function uses. It looks nothing like =E2=80=9Ccorruption of the source=E2=80=9D, whatever you may mean by that. = The error message explains _exactly_ what the compiler is guarding against. Also, thinking about fractions here is irrelevant. The Rust team examined the source of every Rust crate every published on https://crates.io, and found only 5 that even used these characters. With a sample size that small, percentages don=E2=80=99t mean m= uch. https://blog.rust-lang.org/2021/11/01/cve-2021-42574.html > It's the same kind of "solution" like the airport security after 9/11: > because there was a bunch of terrorists, we are all now suspect as > potential terrorists, and for that reason we are constantly delayed > for hours and humiliated by endless frisking. Now I think you are being deliberately insulting. I conclude that your only purpose in this conversation was to troll people or to say no to any solution you didn=E2=80=99t think of yourself. Yours doesn=E2=80=99t even work with `next-error`. Useless. db48x