From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id gGtlIqHWp2V/qAAAe85BDQ:P1 (envelope-from ) for ; Wed, 17 Jan 2024 14:31:13 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id gGtlIqHWp2V/qAAAe85BDQ (envelope-from ) for ; Wed, 17 Jan 2024 14:31:13 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=qZU0GO0X; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=posteo.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1705498273; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=CRIxXhm5NCv5uNn199n98R3aAdjNMKWdheMMCxHyr60=; b=dPO8IVKQlTKFWy/p30/WL6VucH9f4jH/WcB0RAoJQjWwdKnvclqkxiA1ur+9258LvUK8s5 lVe1z6pOLalkfvcYp2h/RbE0XHDPyPob5mhwwZfU4ZLQc6J+ts8SnkJVFI22IvslFhcYAR BOYcBOoKxluGh+GaifZmaQAu18BBpAAxXZydAoRu6pebKvMdSIw3q7nTBFRnKBCM38ak6U 9cdrXQPzC56XGz0gTReoV4gIOYcjcR2LAEoxINqVDIGZ9aDp7fH8RqEiUz0YLL4IRVM4ze 8ZlzxxzrU8BtL3vrh3MSQkli/JSjNPFnkGbBAnF8cKfY8ZfOcFV9rsEVzEbgAQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=qZU0GO0X; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=posteo.net ARC-Seal: i=1; s=key1; d=yhetil.org; t=1705498273; a=rsa-sha256; cv=none; b=P1if6Ll8EcXcnj3rHvkOZuXvqXivhlOjr3dyLz3g96jWPZy5BW/CRrgkkoGffQN6EygZ1G liXr0PXX3wv3tJopNHBD3Qt3jY011oFzApokRCDKnAqRakZQC0HvZcz3GB+8QTkAMz9OEa i45aQLYf7DmqG/cGi71o2fdXVfK6HLDBw5YpaOptdngvyPFaZXykndUjIb0ReH6bFSktWF i33pyN5P+kCqtw3NqIuTl4hYGpoyh7+pnqoD+ocX372fgMryCXcyMAItlFImG6aLTgvllF khHeMPSi08lKQYL2wvQsftnpVZvJMPtvS0VmjaZCSZES176ou6/xWywnPiBofw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4E1DF5A814 for ; Wed, 17 Jan 2024 14:31:13 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rQ56y-0007gH-TB; Wed, 17 Jan 2024 07:33:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rQ56u-0007dY-AB for emacs-orgmode@gnu.org; Wed, 17 Jan 2024 07:33:36 -0500 Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rQ56q-0000Q7-Sn for emacs-orgmode@gnu.org; Wed, 17 Jan 2024 07:33:35 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 098A3240028 for ; Wed, 17 Jan 2024 13:33:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1705494810; bh=IwK/GB6LMpcYJlBImK2NqOs8x0y2foK0brfkUci/cL8=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=qZU0GO0XTA29iBh646U9mmti17Utvrx9owQr242uJa2FucHp+z4YD6Mg0D+It8BDr cUOiqGuDEbnCqu+++BCnGcnl3MrUeHlXfzaY8KsgarsJ45nFs81Qe+zg24Y4of6M9C eqspwWG/uu832rttbZIYr9ytkTnI0fURNVjkTJ4ur5Drcr8y3QheEkvtdIMel5xpqP tP+pez4cuVW6gjXFTxVee/8MTJp6edVR95oG6Gg+WdifPKdmldMY6PgovefTHs9Edj z76f9DZUxYsGZwGosYbTVsSybDx++ZLJDxDnS0vA6h5QbvUVZVucCQz/3FnMtIlyE1 7dNdifZjN6+Ig== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4TFQK50Hsqz9rxP; Wed, 17 Jan 2024 13:33:28 +0100 (CET) From: Ihor Radchenko To: Nathaniel Nicandro Cc: emacs-orgmode Subject: Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements) In-Reply-To: <8734uwhlhj.fsf@gmail.com> References: <874jpuijpc.fsf@gmail.com> <87y1n6igvo.fsf@localhost> <878rev1q0k.fsf@gmail.com> <877cueonkj.fsf@localhost> <87zg6dez93.fsf@gmail.com> <871qjobhwa.fsf@localhost> <877ct5fzt6.fsf@gmail.com> <87a5y1mnj0.fsf@localhost> <87msvcgjgv.fsf@gmail.com> <87le9wq2dg.fsf@localhost> <8734uwhlhj.fsf@gmail.com> Date: Wed, 17 Jan 2024 12:36:43 +0000 Message-ID: <875xzsjfvo.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.67.36.65; envelope-from=yantar92@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -9.89 X-Migadu-Queue-Id: 4E1DF5A814 X-Spam-Score: -9.89 X-Migadu-Scanner: mx11.migadu.com X-TUID: Xrun4ozRKQTP Nathaniel Nicandro writes: > Hello, attached is another updated patch with the following changes: Thanks! > - To tackle the issue discussed previously about highlights spanning > multiple lines (or elements) being removed when a line is modified I > went ahead and used the font-lock-multiline property (see > font-lock-extend-region-multiline and > font-lock-extend-region-functions) across those regions so that on > any edit of one of the lines, the region including all of the ANSI > sequences that affect that line will be re-fontified. This was the > easier solution, but the downside is that it can cause large regions > to be re-fontified when really all we want to do is apply the > highlighting face to a small line change, for example. This is fine. > - To tackle the issue of editing around the invisible ANSI sequences I > left it up to the font-lock process to catch the invisible edits. > Whenever an edit deletes a character of the sequence that renders > the sequence invalid, the font-lock process will reveal the partial > sequence. But I had to limit what was considered a valid ANSI > sequence to get it working in a somewhat acceptable way. > > The problem that I found was that if the buffer contains something > like >=20=20=20 > ^[[43mfoo >=20=20=20 > (where ^[ is the ESC character and can be inserted with "C-q ESC" and > the whole sequence ^[[43m is the ANSI sequence) what was happening was > that deleting into the hidden sequence would leave the region in the > state >=20=20=20 > ^[[43foo >=20=20=20 > and because the end byte of the ANSI sequence can be any character > in the ASCII range [@A-Z[\]^_`a=E2=80=93z{|}~], ^[[43f would still be a > valid ANSI sequence and would be hidden during the fontification > process after the edit. Since `ansi-color-apply-on-region' only > really handles the sequences that end in an m byte, just rendering > all other ones invisible, I limited the ANSI sequences handled by > this patch to be only those sequences that end in m. This way, > after deleting into the sequence like in the above example the > fontification process would not recognize the region as containing > any sequence. The downside to this solution is that sequences that > end in any other end byte won't get conveniently hidden and the > problem still persists if you have text that starts with an m and > you delete into a hidden sequence. Makes sense. We may also make hiding ^[[43foo as customization disabled by default. =20=20=20 > An alternative solution that doesn't constrain the end byte could be > to add in some extra invisible character like a zero width space and > then use something like the `modification-hooks' text property on > the character to signify that a deletion at the boundary between the > sequence and the text should really delete part of the sequence > instead of the zero width space. I haven't really worked out the > details of this, for example how would it be detected which > direction a deletion is coming from, the front or behind, but I'm > throwing it out there to see if there are any other solutions other > people might be aware of for a similar problem. If you want to go in this direction, check out `org-fold-check-before-invisible-edit'. We can unfontify the escape sequence from there and font-lock will re-apply only during the next editing cycle, making the sequence visible temporarily. Not mandatory though. >> P.S. I am not yet commenting on the details in the code. > > Please let me know what you think of this patch and where I should be > focusing my efforts moving forward to get this submitted to Org. I tried to test your newest patch with the example file you provided and I notice two things that would be nice: 1. It is a bit confusing to understand why one or other text is colored without seeing the escape characters. Some customization like `org-link-descriptive' and a command like `org-toggle-link-display' would be nice. I can see some users prefer seeing the escape codes. 2. Using overlays for fontification is problematic. In your example file, table alignment becomes broken when escape sequences are hidden inside overlays: | [31mcell 1 | cell 2 | | cell 3 | cell 4 | looks like | cell 1 | cell 2 | | cell 3 | cell 4 | Using text properties would make table alignment work without adjustments in the org-table.el code. > One thing I would like to start doing is writing some tests for this > feature. It would be great if someone could point me to some tests > that I can peruse so that I can get an idea of how I can go about > writing some of my own. Also, are there any procedures or things I > should be aware of when trying to write my own tests? Check out testing/README file in the Org repository. Unfortunately, we do not yet have any existing tests for font-locking in Org tests. You may still refer to the files in testing/lisp/ to see some example tests. Also, Emacs has built-in library to help writing font-lock tests - faceup.el. You may consider using it. Its top comment also contains a number of references to various tools that could be useful to diagnose font-locking code. --=20 Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at