From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Alan Mackenzie Newsgroups: gmane.emacs.bugs Subject: bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.) Date: Wed, 27 Mar 2024 10:04:03 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19391"; mail-complaints-to="usenet@ciao.gmane.io" Cc: acm@muc.de, Eli Zaretskii , 67455@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Mar 27 11:05:26 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rpQ9u-0004sZ-ME for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 27 Mar 2024 11:05:26 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rpQ9X-00028m-FU; Wed, 27 Mar 2024 06:05:03 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rpQ9W-000287-1E for bug-gnu-emacs@gnu.org; Wed, 27 Mar 2024 06:05:02 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rpQ9V-000268-PS for bug-gnu-emacs@gnu.org; Wed, 27 Mar 2024 06:05:01 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rpQ9V-0000aW-Ii for bug-gnu-emacs@gnu.org; Wed, 27 Mar 2024 06:05:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Alan Mackenzie Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 27 Mar 2024 10:05:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67455 X-GNU-PR-Package: emacs Original-Received: via spool by 67455-submit@debbugs.gnu.org id=B67455.17115338592153 (code B ref 67455); Wed, 27 Mar 2024 10:05:01 +0000 Original-Received: (at 67455) by debbugs.gnu.org; 27 Mar 2024 10:04:19 +0000 Original-Received: from localhost ([127.0.0.1]:35828 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rpQ8o-0000Ye-JE for submit@debbugs.gnu.org; Wed, 27 Mar 2024 06:04:19 -0400 Original-Received: from mail.muc.de ([193.149.48.3]:34923) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rpQ8i-0000Xq-HD for 67455@debbugs.gnu.org; Wed, 27 Mar 2024 06:04:17 -0400 Original-Received: (qmail 52467 invoked by uid 3782); 27 Mar 2024 11:04:05 +0100 Original-Received: from acm.muc.de (pd953a0c3.dip0.t-ipconnect.de [217.83.160.195]) (using STARTTLS) by colin.muc.de (tmda-ofmipd) with ESMTP; Wed, 27 Mar 2024 11:04:05 +0100 Original-Received: (qmail 20666 invoked by uid 1000); 27 Mar 2024 10:04:03 -0000 Content-Disposition: inline In-Reply-To: X-Submission-Agent: TMDA/1.3.x (Ph3nix) X-Primary-Address: acm@muc.de X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:282130 Archived-At: Hello, Stefan. On Tue, Mar 26, 2024 at 16:30:06 -0400, Stefan Monnier wrote: > > We now have two distinct uses of SWPs: providing warning source locations > > to the compiler (where we want to keep the position as long as possible) > > and providing position information for the doc string (where we want to > > strip the position from the symbol ASAP, to avoid trying to use the SWP > > when we need a plain symbol). If both of these occur together, we want > > to keep the SWP. > I think I'm beginning to understand. So in the "load from source case", > some of your symbols are SWPs and you want to turn them into bare > symbols "on the fly" during macro-expansion rather than via a separate > "strip" phase, .... More precisely, to @dfn{posify} their containing forms by writing the position information into their doc strings. We do this in the byte compilation case, too. The difference is that in the "load from source" case we want to strip the SWP, in byte compilation, we don't. > .... so you want the macro expansion to know whether it's done > for "load from source" or for some other purpose and you use > `byte-compile-in-progress` as a proxy for that information. > Is that it? More or less. But byte-compile-in-progress isn't a proxy, it's the prime criterion for deciding. > If so, (and if the stripping happens within macros), then indeed passing > it as a separate argument through all the recursive calls to > `macroexp--expand-all` would be cumbersome. But I suggest you use > another name for that(e.g. `macroexp-strip-position`) so the > intention is made more clear. I don't think that is a good name. The byte compiler has no business setting "internal" variables for the posification processing. Instead it should announce it's running and expect the posification to respect that. I think byte-compile-in-progress is a good name for this. > Better yet: to avoid the problem of dynamic scope extending "too far" > (i.e. accidentally applying to nested loads/evals/compile/...), you > could put the relevant info into `macroexpand-all-environment`. > [ That var is also dynamically bound, but we're already careful to > rebind it when needed so it doesn't apply to nested uses > of macroexpansion. ] That variable is only loaded in the 17th loaded Lisp file. The new facility should be working at the earliest stages of loading Lisp, as it does at the moment. Besides, macroexpand-all-environment is not documented anywhere, what it is, what it's for, etc. > >> > By "currently", I mean that a defining form such as defun or defvar has > >> > commenced, but not yet terminated; its functions currently occupy stack > >> > frames. > >> So you mean we're inside `Fdefalias` or `Fdefvar_1`? > > Yes, or inside a macro (defun, defmacro, ...) which expands to a > > defalias. > These are *very* different times: `Fdefalias` and `Fdefvar_1` are > executed long after macroexpansion. And they're small C functions which > run almost no external code at all, so they "occupy stack frames" only > for a very short time. I think we were talking about the handling of defining-symbol. It has a valid binding outside of macros such as defun, and this binding is used to posify the containing form. > My crystal ball suggests that "currently" may be the wrong way to think > about it: maybe instead of thinking of "when" (as in "during the > definition of function FOO") what you're looking for might be "where" > (as in "within the body of FOO"). > [ That's the same difference as the difference between dynamic and > static scoping. ] I'm having trouble understanding what you're saying, here. > If my crystal ball is right, then the better place to put that info is > probably `macroexpand-all-environment`. See above. > > Ideally, I would like to have bound defining-symbol inside defun. > (defmacro my-defun (name args &rest body) > `(cl-macrolet ((defining-symbol () '',name)) > (defun ,name ,args ,@body))) > (my-defun my-foo (x) (list x (defining-symbol))) > (symbol-function 'my-foo) > ==> #f(lambda (x) [t] (list x 'my-foo)) > `cl-macrolet` uses `macroexpand-all-environment` for that. cl-macs gets loaded far too late for such an approach to be useful. > > Fload uses read-positioning-DEFINED-symbols, as contrasted with the > > compiler, which uses read-positioning-symbols. r-p-d-s positions only > > lambdas and NAMEs. r-p-s positions all symbols except nil. > I think I'm beginning to understand (I guess I was struggling with your > use of "position" as a verb for some reason which made me think that > symbols were being moved rather than gaining position information). Sorry about that. > So, IIUC you use `read-positioning-DEFINED-symbols` instead of > `read-positioning-symbols` because it's cheaper? No, because it does the Right Thing. > Do you have rough numbers comparing the cost of `read`, > `read-positioning-symbols`, and `read-positioning-DEFINED-symbols`? No, but they will be very close to eachother (and very cheap) since they use the same engine, read0 (in lread.c). Each of them will be one or two orders of magnitude faster than emulating them in Lisp. > Also, IIUC you don't have a separate phase to strip the SWPs when > loading from source, but instead you strip them as you "consume" their > info during macroexpansion. If so, how/when/where do you strip the > false positives that may occur inside quoted data or in code like: > (defmacro foo (lambda bar) ...) > (defmacro foo (defun bar) ...) > (let* ((lambda foo) > (defun bar)) > ...) There's a pcase arm right at the end of macroexp--expand-all which strips SWPs of their positions. Recursing through macroexp--all-forms will eventually hit this pcase arm for these lambdas. > > Ah, right. I hadn't considered this before. The changes are by their > > very nature essentially complicated and difficult to understand. > [ Hmm... maybe not the best salespitch. ] ;-) It's the truth, though. > >> I think you can simply wait to add the entry to > >> `macro-declarations-alist` until a later time, so the `defining-symbol` > >> thingies will be ignored during the early bootstrap and once we have > >> more infrastructure in place we can then register the handler on > >> `macro-declarations-alist`. > > This will not be simpler. It would involve re-evaluating defun, then > > compensating for all the functions up to now whose NAMEs had been read > > without positions. > Not at all. Those will remain without position, but only in > `src/bootstrap-emacs`. This would be a Bad Thing. The current code is active right after loading byte-run. > In the real `src/emacs` they will get the position because they'll come > from the `.el[cn]` file and by the time we get compile those files > `macro-declarations-alist` will be fully populated. The understanding we reached in November was that loading from source files would be handled, too. > > There is unavoidable conplexity, here. > I'm definitely not convinced. I suspect you've been asking yourself > "can it be made simpler" and you may indeed then convince yourself that > the answer is no, because of assumptions you don't reconsider. > Try instead to think about "what would it take to remove that complexity?". > > I see things somewhat differently. We shouldn't increase the debugging > > burden even on "expert users". > Yet it's imposing more complexity (and hence more debugging burden) on > those same expert users. 🙁 That's a fairly difficult philosophical question. Do we provide full functionality at the cost of more (difficult) source code, or do we restrict the functioality to keep the source simpler? I think with Emacs we usually go with the first alternative. > Stefan -- Alan Mackenzie (Nuremberg, Germany).