From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#70077: An easier way to track buffer changes Date: Fri, 29 Mar 2024 12:15:53 -0400 Message-ID: Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="31411"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Nicolas Goaziou , Ihor Radchenko , Alan Mackenzie , =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= , Alan Zimmerman , =?UTF-8?Q?Fr=C3=A9d=C3=A9ric?= Bour , Phillip Lord , Stephen Leake , Yuan Fu , Qiantan Hong , monnier@iro.umontreal.ca To: 70077@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri Mar 29 17:17:20 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rqEus-0007uZ-Jd for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 29 Mar 2024 17:17:18 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rqEuh-00018E-2Y; Fri, 29 Mar 2024 12:17:07 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rqEue-00017m-Hv for bug-gnu-emacs@gnu.org; Fri, 29 Mar 2024 12:17:04 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rqEue-0008Lz-A0 for bug-gnu-emacs@gnu.org; Fri, 29 Mar 2024 12:17:04 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rqEub-0001kr-Rh; Fri, 29 Mar 2024 12:17:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: mail@nicolasgoaziou.fr, yantar92@gmail.com, acm@muc.de, joaotavora@gmail.com, alan.zimm@gmail.com, frederic.bour@lakaban.net, phillip.lord@russet.org.uk, stephen_leake@stephe-leake.org, casouri@gmail.com, qhong@alum.mit.edu, monnier@iro.umontreal.ca, bug-gnu-emacs@gnu.org Resent-Date: Fri, 29 Mar 2024 16:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 70077 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch X-Debbugs-Original-To: bug-gnu-emacs@gnu.org X-Debbugs-Original-Xcc: Nicolas Goaziou , Ihor Radchenko , Alan Mackenzie , =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= , Alan Zimmerman , =?UTF-8?Q?Fr=C3=A9d=C3=A9ric?= Bour , Phillip Lord , Stephen Leake , Yuan Fu , Qiantan Hong , monnier@iro.umontreal.ca Original-Received: via spool by submit@debbugs.gnu.org id=B.17117289956682 (code B ref -1); Fri, 29 Mar 2024 16:17:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 29 Mar 2024 16:16:35 +0000 Original-Received: from localhost ([127.0.0.1]:43253 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rqEu9-0001jf-Tb for submit@debbugs.gnu.org; Fri, 29 Mar 2024 12:16:35 -0400 Original-Received: from lists.gnu.org ([2001:470:142::17]:42846) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rqEu6-0001jP-IY for submit@debbugs.gnu.org; Fri, 29 Mar 2024 12:16:31 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rqEty-00013T-QT for bug-gnu-emacs@gnu.org; Fri, 29 Mar 2024 12:16:22 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rqEtv-0008Hr-V8 for bug-gnu-emacs@gnu.org; Fri, 29 Mar 2024 12:16:22 -0400 Original-Received: from pmg2.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 7209080AD9 for ; Fri, 29 Mar 2024 12:16:17 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1711728975; bh=HWHFnxY6gTAT9y6U3RemvbTpKbH0HnPlbAz3V/Y8dTo=; h=From:To:Subject:Date:From; b=nkz2zDG9v105OpPhH95wm9azILeG9I6q0U9ZGQgj3e1YduybOn/HOaJiaGSR+xQLw KZl34Cxt+fBsW0WwUMnzYSO2U+IoEwhQglRGZigpM+yAxOtNycuhwt3FfyyzEBGkIX Uqz1i03ow37zetuz8rI5f9QQwmSZlU4FwYLXyW2AV5uLAfX8FKh+VVnrVhAo+EU0WD CmsltX7dYZacV5p2MmgdtJ94UvYuMtkGdnitpFd82wwvPt9aRGfhNXYiBo7KszT4xT mF7sK5jsycsgURUz2m1l9DAbTbO+VbXU8xjescn6skKRs+4O/vm9iL1yKhJEYdj9UN tv4/xm3Fcv70A== Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 1C30A8009D for ; Fri, 29 Mar 2024 12:16:15 -0400 (EDT) Original-Received: from alfajor (unknown [23.233.149.155]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 06D8D12077C for ; Fri, 29 Mar 2024 12:16:15 -0400 (EDT) Received-SPF: pass client-ip=132.204.25.50; envelope-from=monnier@iro.umontreal.ca; helo=mailscanner.iro.umontreal.ca X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:282305 Archived-At: --=-=-= Content-Type: text/plain Tags: patch Our `*-change-functions` hook are fairly tricky to use right. Some of the issues are: - before and after calls are not necessarily paired. - the beg/end values don't always match. - there can be thousands of calls from within a single command. - these hooks are run at a fairly low-level so there are things they really shouldn't do, such as modify the buffer or wait. - the after call doesn't get enough info to rebuild the before-change state, so some callers need to use both before-c-f and after-c-f (and then deal with the first two points above). The worst part is that those problems occur rarely, so many coders don't see it at first and have to learn them the hard way, sometimes forcing them to rethink their original design. So I think we should provide something simpler. I attached a proof-of-concept API which aims to do that, with the following entry points: (defun track-changes-register ( signal) "Register a new tracker and return a new tracker ID. SIGNAL is a function that will be called with no argument when the current buffer is modified, so that we can react to the change. Once called, SIGNAL is not called again until `track-changes-fetch' is called with the corresponding tracker ID." (defun track-changes-unregister (id) "Remove the tracker denoted by ID. Trackers can consume resources (especially if `track-changes-fetch' is not called), so it is good practice to unregister them when you don't need them any more." (defun track-changes-fetch (id func) "Fetch the pending changes. ID is the tracker ID returned by a previous `track-changes-register'. FUNC is a function. It is called with 3 arguments (BEGIN END BEFORE) where BEGIN..END delimit the region that was changed since the last time `track-changes-fetch' was called and BEFORE is a string containing the previous content of that region. If no changes occurred since the last time, FUNC is not called and we return nil, otherwise we return the value returned by FUNC, and re-enable the TRACKER corresponding to ID." It's not meant as a replacement of the existing hooks since it doesn't try to accommodate some uses such as those that use before-c-f to implement a finer-grained form of read-only text. The driving design was: - Try to provide enough info such that it is possible and easy to maintain a copy of the buffer simply by applying the reported changes. E.g. for uses such as `eglot.el` or `crdt.el`. - Make the API less synchronous: take care of combining small changes into larger ones, and let the clients decide when they react to changes. If you're in the Cc, it's because I believe you have valuable experience with those hooks, so I'd be happy to hear your thought about whether you think this would indeed (have) be(en) better than what we have. Stefan --=-=-= Content-Type: text/x-emacs-lisp; charset=iso-8859-1 Content-Disposition: attachment; filename=track-changes.el Content-Transfer-Encoding: quoted-printable ;;; track-changes.el --- API to react to buffer modifications -*- lexical-= binding: t; -*- ;; Copyright (C) 2024 Free Software Foundation, Inc. ;; Author: Stefan Monnier ;; This file is part of GNU Emacs. ;; GNU Emacs is free software: you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; GNU Emacs is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs. If not, see . ;;; Commentary: ;; This library is a layer of abstraction above `before-change-functions' ;; and `after-change-functions' which takes care of accumulating changes ;; until a time when its client finds it convenient to react to them. ;; It provides the following operations: ;; ;; (track-changes-register SIGNAL) ;; (track-changes-fetch ID FUNC) ;; (track-changes-unregister ID) ;; ;; A typical use case might look like: ;; ;; (defvar my-foo--change-tracker nil) ;; (define-minor-mode my-foo-mode ;; "Fooing like there's no tomorrow." ;; (if (null my-foo-mode) ;; (when my-foo--change-tracker ;; (track-changes-unregister my-foo--change-tracker) ;; (setq my-foo--change-tracker nil)) ;; (unless my-foo--change-tracker ;; (setq my-foo--change-tracker ;; (track-changes-register ;; (lambda () ;; (track-changes-fetch ;; my-foo--change-tracker ;; (lambda (beg end before) ;; ..DO THE THING..)))))))) ;;; Code: ;; FIXME: Try and do some sanity-checks (e.g. looking at `buffer-size'), ;; to detect if/when we somehow missed some changes. ;; FIXME: The API doesn't offer an easy way to signal a "full resync" ;; kind of change, as might be needed if we lost changes. (require 'cl-lib) (cl-defstruct (track-changes--tracker (:noinline t) (:constructor nil) (:constructor track-changes--tracker ( signal state))) ( signal nil :read-only t) state) (cl-defstruct (track-changes--state (:noinline t) (:constructor nil) (:constructor track-changes--state ())) (beg (point-max)) (end (point-min)) (bbeg (point-max)) ;BEG of the BEFORE string, (bend (point-min)) ;END of the BEFORE string. (before nil) (next nil)) (defvar-local track-changes--trackers ()) (defvar-local track-changes--clean-trackers ()) (defvar-local track-changes--state nil) (defun track-changes-register ( signal) "Register a new tracker and return a new tracker ID. SIGNAL is a function that will be called with no argument when the current buffer is modified, so that we can react to the change. Once called, SIGNAL is not called again until `track-changes-fetch' is called with the corresponding tracker ID." ;; FIXME: Add an optional arg to choose between `funcall' and `funcall-la= ter'? (track-changes--clean-state) (add-hook 'before-change-functions #'track-changes--before nil t) (add-hook 'after-change-functions #'track-changes--after nil t) (let ((tracker (track-changes--tracker signal track-changes--state))) (push tracker track-changes--trackers) (push tracker track-changes--clean-trackers) tracker)) (defun track-changes-unregister (id) "Remove the tracker denoted by ID. Trackers can consume resources (especially if `track-changes-fetch' is not called), so it is good practice to unregister them when you don't need them any more." (unless (memq id track-changes--trackers) (error "Unregistering a non-registered tracker: %S" id)) (setq track-changes--trackers (delq id track-changes--trackers)) (setq track-changes--clean-trackers (delq id track-changes--clean-tracker= s)) (when (null track-changes--trackers) (setq track-changes--state nil) (remove-hook 'before-change-functions #'track-changes--before t) (remove-hook 'after-change-functions #'track-changes--after t))) (defun track-changes--clean-p () (null (track-changes--state-before track-changes--state))) (defun track-changes--clean-state () (cond ((null track-changes--state) ;; No state has been created yet. Do it now. (setq track-changes--state (track-changes--state))) ((track-changes--clean-p) nil) (t ;; FIXME: We may be in-between a before-c-f and an after-c-f, so we ;; should save some of the current buffer in case an after-c-f comes ;; before a before-c-f. (let ((new (track-changes--state))) (setf (track-changes--state-next track-changes--state) new) (setq track-changes--state new))))) (defun track-changes--before (beg end) (cl-assert track-changes--state) (cl-assert (<=3D beg end)) (if (track-changes--clean-p) (progn (setf (track-changes--state-before track-changes--state) (buffer-substring-no-properties beg end)) (setf (track-changes--state-bbeg track-changes--state) beg) (setf (track-changes--state-bend track-changes--state) end)) (cl-assert (save-restriction (widen) (<=3D (point-min) (track-changes--state-bbeg track-changes--state) (track-changes--state-bend track-changes--state) (point-max)))) (when (< beg (track-changes--state-bbeg track-changes--state)) (let* ((old-bbeg (track-changes--state-bbeg track-changes--state)) ;; To avoid O(N=B2) behavior when faced with many small change= s, ;; we copy more than needed. (new-bbeg (min (max (point-min) (- old-bbeg (length (track-changes--state-before track-changes--state)))) beg))) (setf (track-changes--state-bbeg track-changes--state) beg) (cl-callf (lambda (old new) (concat new old)) (track-changes--state-before track-changes--state) (buffer-substring-no-properties new-bbeg old-bbeg)))) (when (< (track-changes--state-bend track-changes--state) end) (let* ((old-bend (track-changes--state-bend track-changes--state)) ;; To avoid O(N=B2) behavior when faced with many small change= s, ;; we copy more than needed. (new-bend (max (min (point-max) (+ old-bend (length (track-changes--state-before track-changes--state)))) end))) (setf (track-changes--state-bend track-changes--state) end) (cl-callf concat (track-changes--state-before track-changes--state) (buffer-substring-no-properties old-bend new-bend)))))) (defun track-changes--after (beg end len) (cl-assert track-changes--state) (cl-assert (track-changes--state-before track-changes--state)) (let ((offset (- (- end beg) len))) (cl-incf (track-changes--state-bend track-changes--state) offset) (cl-assert (save-restriction (widen) (<=3D (point-min) (track-changes--state-bbeg track-changes--state) beg end (track-changes--state-bend track-changes--state) (point-max)))) ;; Note the new changes. (when (< beg (track-changes--state-beg track-changes--state)) (setf (track-changes--state-beg track-changes--state) beg)) (cl-callf (lambda (old-end) (max end (+ old-end offset))) (track-changes--state-end track-changes--state))) (cl-assert (<=3D (track-changes--state-bbeg track-changes--state) (track-changes--state-beg track-changes--state) beg end (track-changes--state-end track-changes--state) (track-changes--state-bend track-changes--state))) (while track-changes--clean-trackers (let ((tracker (pop track-changes--clean-trackers))) ;; FIXME: Use `funcall'? (funcall-later (track-changes--tracker-signal tracker) ())))) (defun track-changes-fetch (id func) "Fetch the pending changes. ID is the tracker ID returned by a previous `track-changes-register'. FUNC is a function. It is called with 3 arguments (BEGIN END BEFORE) where BEGIN..END delimit the region that was changed since the last time `track-changes-fetch' was called and BEFORE is a string containing the previous content of that region. If no changes occurred since the last time, FUNC is not called and we return nil, otherwise we return the value returned by FUNC, and re-enable the TRACKER corresponding to ID." (let ((beg nil) (end nil) (before nil) (states ())) ;; We want to combine the states from most recent to oldest, ;; so reverse them. (let ((state (track-changes--tracker-state id))) (while state (push state states) (setq state (track-changes--state-next state)))) (when (null (track-changes--state-before (car states))) (cl-assert (eq (car states) track-changes--state)) (setq states (cdr states))) (if (null states) (progn (cl-assert (memq id track-changes--clean-trackers)) nil) (dolist (state states) (let ((prevbbeg (track-changes--state-bbeg state)) (prevbend (track-changes--state-bend state)) (prevbefore (track-changes--state-before state))) (if (not before) (progn ;; This is the most recent change. Just initialize the var= s. (setq beg (track-changes--state-beg state)) (setq end (track-changes--state-end state)) (setq before prevbefore) (unless (and (=3D beg prevbbeg) (=3D end prevbend)) (setq before (substring before (- beg (track-changes--state-bbeg state)) (- (length before) (- (track-changes--state-bend state) end)))))) ;; FIXME: When merging "states", we disregard the `beg/end' ;; in favor of `bbeg/bend' which also works but is conservative. (let ((endb (+ beg (length before)))) (when (< prevbbeg beg) (setq before (concat (buffer-substring-no-properties prevbbeg beg) before)) (setq beg prevbbeg) (cl-assert (=3D endb (+ beg (length before))))) (when (< endb prevbend) (let ((new-end (+ end (- prevbend endb)))) (setq before (concat before (buffer-substring-no-properties end new-end))) (setq end new-end) (cl-assert (=3D prevbend (+ beg (length before)))) (setq endb (+ beg (length before))))) (cl-assert (<=3D beg prevbbeg prevbend endb)) ;; The `prevbefore' is covered by the new one. (setq before (concat (substring before 0 (- prevbbeg beg)) prevbefore (substring before (- (length before) (- endb prevbend))))))))) (cl-assert (<=3D (point-min) beg end (point-max))) ;; Clean the state of the tracker before calling `func', in case ;; `func' performs buffer modifications. (track-changes--clean-state) ;; Update the tracker's state before running `func' so we don't risk ;; mistakenly replaying the changes in case `func' exits non-locally. (setf (track-changes--tracker-state id) track-changes--state) (unwind-protect (funcall func beg end before) ;; Re-enable the tracker's signal only after running `func', so ;; as to avoid recursive invocations. (cl-pushnew id track-changes--clean-trackers))))) (defmacro with-track-changes (id vars &rest body) (declare (indent 2) (debug (form sexp body))) `(track-changes-fetch ,id (lambda ,vars ,@body))) =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 (provide 'track-changes) ;;; track-changes.el end here. --=-=-=--