From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Alan Mackenzie Newsgroups: gmane.emacs.bugs Subject: bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.) Date: Tue, 26 Mar 2024 09:48:49 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="21780"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acm@muc.de, Eli Zaretskii , 67455@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Mar 26 14:49:15 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rp7Ax-0005Ww-AF for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 26 Mar 2024 14:49:15 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rp7An-00026a-86; Tue, 26 Mar 2024 09:49:05 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rp7Al-00026I-BC for bug-gnu-emacs@gnu.org; Tue, 26 Mar 2024 09:49:03 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rp7Al-0008UP-0T for bug-gnu-emacs@gnu.org; Tue, 26 Mar 2024 09:49:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rp7Ak-0000vr-Fy for bug-gnu-emacs@gnu.org; Tue, 26 Mar 2024 09:49:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Alan Mackenzie Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 26 Mar 2024 13:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67455 X-GNU-PR-Package: emacs Original-Received: via spool by 67455-submit@debbugs.gnu.org id=B67455.17114609423573 (code B ref 67455); Tue, 26 Mar 2024 13:49:02 +0000 Original-Received: (at 67455) by debbugs.gnu.org; 26 Mar 2024 13:49:02 +0000 Original-Received: from localhost ([127.0.0.1]:33136 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rp7Aj-0000vY-51 for submit@debbugs.gnu.org; Tue, 26 Mar 2024 09:49:01 -0400 Original-Received: from mail.muc.de ([193.149.48.3]:56714) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rp7Ag-0000v7-9V for 67455@debbugs.gnu.org; Tue, 26 Mar 2024 09:49:00 -0400 Original-Received: (qmail 94037 invoked by uid 3782); 26 Mar 2024 10:48:50 +0100 Original-Received: from acm.muc.de (p4fe15213.dip0.t-ipconnect.de [79.225.82.19]) (using STARTTLS) by colin.muc.de (tmda-ofmipd) with ESMTP; Tue, 26 Mar 2024 10:48:50 +0100 Original-Received: (qmail 18562 invoked by uid 1000); 26 Mar 2024 09:48:49 -0000 Content-Disposition: inline In-Reply-To: X-Submission-Agent: TMDA/1.3.x (Ph3nix) X-Primary-Address: acm@muc.de X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:282085 Archived-At: Hello, Stefan. On Mon, Mar 25, 2024 at 18:10:11 -0400, Stefan Monnier wrote: > >> That sounds rather ugly. I'd rather first try and define precisely > >> what is we mean by "compilation in progress". > > That the byte compiler is active, and all symbols (bar nil) get > > positioned. > That is much too vague to be usable. Which symbols are we > talking about? What do we mean by "active"? The symbols read from the source code being compiled by the byte compiler. By "active" I mean that a byte compilation has been started, is not yet complete, and hasn't been temporarily suspended (e.g. for loading another file with (require 'foo)). > I think we need to describe it in terms that link the symbol with code > in bytecomp.el (not based on time, but based on function bytecomp-FOO > calling BAR calling BAZ ... touching that symbol). > Note that I also have no idea why we need to care about whether we're in > the compiled case or the non-compiled case, so maybe my questions are > "XY" style problems. We now have two distinct uses of SWPs: providing warning source locations to the compiler (where we want to keep the position as long as possible) and providing position information for the doc string (where we want to strip the position from the symbol ASAP, to avoid trying to use the SWP when we need a plain symbol). If both of these occur together, we want to keep the SWP. This is the purpose of byte-compile-in-progress. > > An alternative might be to pass an &optional boolean argument meaning > > "preserve positions on symbols". > >From where to where should we pass that argument? > [ And yes, explicit arguments might be preferable unless the > call-trace is too long for it to be practical or unless some of the > functions along that call trace can't be changed easily to take such an > extra argument. ] To macroexp--expand-all from wherever. Here (L423 of macroexp.el) is where the position gets stripped from the SWP in the non compilation case. > BTW, do you really mean "preserve" (meaning that the symbols were sympos > to start with)? Indeed, yes. > >> I see the same problem with: > >> DEFVAR_LISP ("defining-symbol", Vdefining_symbol, > >> doc: /* The symbol currently being defined by a defining form. > >> This variable is bound in the read-eval-print loop and certain > >> high-level functions in the byte compiler. It is set to a value by > >> functions and macros such as `defun', `defmacro', and `defvar'. */); > >> Lots and lots of things can happen "during the definition" of a form, > >> including definition of lots of other forms. So I think we'd need to > >> define much more precisely what you meant by "currently". > >> In addition, a definition is "intemporal" (it's declarative), so > >> "currently being defined" is almost like an oxymoron. > > By "currently", I mean that a defining form such as defun or defvar has > > commenced, but not yet terminated; its functions currently occupy stack > > frames. > So you mean we're inside `Fdefalias` or `Fdefvar_1`? Yes, or inside a macro (defun, defmacro, ...) which expands to a defalias. Ideally, I would like to have bound defining-symbol inside defun. But this would have lost the binding at the end of defun, before evaluating the defalias. It was a tricky problem which I think has been solved. > >> I'm trying to understand your code, but I clearly lack a high-level > >> overview of the approach you decided to takes, so I don't understand > >> what's going on there. > > Sorry about that. A quick summary: defined symbols (and lambda) get > > positioned by the new reader function read-positioning-defined symbols. > > The new declare clause defining-symbol marks a macro such as defun or > > cl-defgeneric as a macro which defines such symbols. > > The conversion of these SWPs into position structures in doc strings > > happens at macro expansion time, through byte-run-posify-lambda-form. > So, IIUC > (defmacro defun (name args docstring &rest body) > (declare (defining-symbol 1)) > ...) > is akin to: > (defmacro defun (name args docstring &rest body) > (setq docstring (add-pos-to-docstring > (symbol-with-pos-pos name) docstring)) > ...) > ? Pretty much, yes. (declare (defining-symbol name docstring)) also informs the reader that NAME is to be positioned when in (defun NAME ....). > >> If both, could you split it into two, then? > > I'm not sure that would be possible or sensible - both use a common > > approach. > So, IIUC a first part of the change was to make `load` use > `read-positioning-symbols` just like the compiler? Fload uses read-positioning-DEFINED-symbols, as contrasted with the compiler, which uses read-positioning-symbols. r-p-d-s positions only lambdas and NAMEs. r-p-s positions all symbols except nil. > >> AFAICT doing it only for compiled functions should be significantly > >> simpler than for interpreted functions, so it would be a good > >> stepping stone. > > The work has already been done, and there is working code. Just as a > > matter of interest, the branch runs the test suite without errors (not > > counting "expensive" tests ). > I'm not talking about a separate branch. I'm talking about splitting > your changes into understandable commits. Ah, right. I hadn't considered this before. The changes are by their very nature essentially complicated and difficult to understand. > >> On the cosmetic side, you have way too much code in `byte-run.el`. > >> I think most of this code can be moved elsewhere, e.g. somewhere where > >> backquote can be used > > Yes, I noticed this, too. A lot of the bulk is for diagnostic functions > > for SWPs, and these can eventually be deleted. Or possibly moved into a > > new file with-pos.el to be loaded before byte-run.el. > I don't like the idea of adding another file before byte-run.el. Neither do I, but it's a possibility. > > byte-run--posify-defining-symbol, the function with the extreme hand > > expansion of backquotes is used as a declare clause handler, and is > > needed by defun. Hence it couldn't really be moved to after the loading > > of backquote.el. > I think you can simply wait to add the entry to > `macro-declarations-alist` until a later time, so the `defining-symbol` > thingies will be ignored during the early bootstrap and once we have > more infrastructure in place we can then register the handler on > `macro-declarations-alist`. This will not be simpler. It would involve re-evaluating defun, then compensating for all the functions up to now whose NAMEs had been read without positions. There is unavoidable conplexity, here. We need defun to build backquote, byte-run--posify-defining-symbol to build defun, and so we need to write b-r--p-d-s without backquote. All that could be done is to shift the complexity to a different arm of that dependency triangle. > > There are some additional functions which batch-posify functions and > > variables defined before the posification mechanism is in place. This > > must be done ASAP, for the benefit of backtraces in early bootstrap. > That complexifies the early bootstrap code. I'd rather keep that code > simpler. During early bootstrap, all those functions are interpreted > and I can't remember ever having difficulty tracking the origin of those > interpreted lambdas during early bootstrap, so I'm rather opposed to > such complexity just for the sake of maybe occasionally hypothetically > giving a bit of help to the 3 of us who have to deal with > those problems. Well all I can say here is that the lack of special cases here has been most helpful in debugging the current code. It's possible (?likely) that somebody will need to look at it again, sometime. > I see functions-with-pos as something mostly targeted at "end users" who > aren't expert enough to figure out the origin of anonymous functions by > looking at the disassembly of the bytecode and taking an educated guess. I see things somewhat differently. We shouldn't increase the debugging burden even on "expert users". My view is that debugging Lisp in Emacs is too difficult and tedious, and can be improved. debug-early.el and getting backtraces from redisplay errors are two already implemented such improvements. > Stefan -- Alan Mackenzie (Nuremberg, Germany).