From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#51982: Erroneous handling of local variables in byte-compiled nested lambdas Date: Tue, 30 Nov 2021 17:41:51 -0500 Message-ID: References: <87y25jo2q1.fsf@web.de> <29C3A3F8-CD9F-4AF2-A731-3304FC30E380@acm.org> <87wnl23pnd.fsf@web.de> <59A729EF-C4D4-47EB-9ADC-19FE8EBE7F10@acm.org> <877dd0bi17.fsf@web.de> Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11783"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: Michael Heerdegen , Paul Pogonyshev , 51982@debbugs.gnu.org To: Mattias =?UTF-8?Q?Engdeg=C3=A5rd?= Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Nov 30 23:43:28 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1msBqS-0002ru-0C for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 30 Nov 2021 23:43:28 +0100 Original-Received: from localhost ([::1]:50134 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1msBqQ-0007eT-GG for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 30 Nov 2021 17:43:26 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:33352) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1msBq5-0007e9-Np for bug-gnu-emacs@gnu.org; Tue, 30 Nov 2021 17:43:06 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:59449) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1msBq2-0006IC-9y for bug-gnu-emacs@gnu.org; Tue, 30 Nov 2021 17:43:05 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1msBq1-0000zC-QE for bug-gnu-emacs@gnu.org; Tue, 30 Nov 2021 17:43:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 30 Nov 2021 22:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51982 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 51982-submit@debbugs.gnu.org id=B51982.16383121243720 (code B ref 51982); Tue, 30 Nov 2021 22:43:01 +0000 Original-Received: (at 51982) by debbugs.gnu.org; 30 Nov 2021 22:42:04 +0000 Original-Received: from localhost ([127.0.0.1]:42762 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1msBp5-0000xw-Ng for submit@debbugs.gnu.org; Tue, 30 Nov 2021 17:42:04 -0500 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:53865) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1msBp3-0000xP-7U for 51982@debbugs.gnu.org; Tue, 30 Nov 2021 17:42:02 -0500 Original-Received: from pmg1.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 20A5D100164; Tue, 30 Nov 2021 17:41:55 -0500 (EST) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id BF3F210000D; Tue, 30 Nov 2021 17:41:52 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1638312112; bh=Hlu2sMLfwIK6EWBBUxUhtUOt36Q8eiYHsXn7Crm0ns8=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=cpcIxPVv4HMe2CTkOYPSKZ/rLt1zVHx+7EHUQWHLwXgD4IDaR5+BR1bgY/+hcwww8 +Zqry2Ez8V0Ta6XxPYTpNobcO2L7B6m0As7XlJcNIx6ArB4PPDSIqT838U4Xp2agM4 7NtJrYvRdthWI80p5Hnd3pC5Iy8vYgy0iPTDOh2O9L6QpdrrBEFkWEuBBDk27YzstY k3RxGLC8OU9Tehgrm1mpgNawAg36PRpks1igjx8HXVDuXwNYGOtvnxaMA58gpW9s3h d+oId+4VGd/P6ULL9sbVMlCMqfcnNVW+hanStIlvsoVnZZ81VsgB8Q/R1LK7x6+jii eu61/pBEb2XBg== Original-Received: from alfajor (lechon.iro.umontreal.ca [132.204.27.242]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 9A9B312069F; Tue, 30 Nov 2021 17:41:52 -0500 (EST) In-Reply-To: ("Mattias =?UTF-8?Q?Engdeg=C3=A5rd?="'s message of "Tue, 30 Nov 2021 18:01:59 +0100") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:221182 Archived-At: > (Really want to get rid of `let*`!) FWIW, we could get rid of it in `cconv-convert`. That would have less impact than doing it in macroexpansion. [ We could also force dynamically-scoped code to go through (a neutered version of) cconv.el , so that bytecomp.el and byte-opt.el can presume that `let*` doesn't exist any more. ] Tho maybe we'd want to eliminate `let` instead. ;-) BTW, have you checked the impact on byte-code quality? >> These two tests are identical aren't they? > No, they exercise different code paths (let and let*). Then that deserves a comment ;-) >> Looks good (better than patch A). > > And here I was prepared to apply patch A since it's slightly more > conservative and it seems to be a rare problem anyway. > I've now split the patches in a more sensible (and easily reviewed) way: = the > first corresponds to patch A, and the second is the diff to B. Take a sec= ond > look before making up your mind. > >> You say "On the other hand, patch B does abuse the cconv data structures >> a little (but it works!)" so the code should say something about >> this abuse. A least I failed to see where the abuse lies. > > There are comments and doc strings such as > > EXTEND is a list of variables which might need to be accessed even > from places where they are shadowed, because some part of ENV causes > them to be used at places where they originally did not > directly appear. > > but with the B patch we put things into `extend` that are not strictly > variables but (international-get-closed-var N). See below, I think we don't need to put them there. > Similarly, `env` has entries like (VAR . (apply-partially F ARG1 ARG2 ..)) > where the ARGi are always treated as variables but now they can be access > forms as well. I don't think the current code assumes that ARGs are vars here. You're probably right that it used to be the case and it's not any more, but that shouldn't cause problems. The risk I can see is if one of those ARGs is an expression which refers to a var which gets shadowed, in which case `cconv--remap-llv` won't rewrite it the way it should. But I think with your code ARG will either be a simple var or something of the form (internal-get-closed-var N) so we should be safe. > @@ -304,6 +304,22 @@ cconv--convert-funcbody > `(,@(nreverse special-forms) ,@(macroexp-unprogn body)))) > funcbody))) >=20=20 > +(defun cconv--lifted-arg (var env) > + "The argument to use for VAR in =CE=BB-lifted calls according to ENV." > + (let ((mapping (cdr (assq var env)))) > + (pcase-exhaustive mapping > + (`(internal-get-closed-var . ,_) > + ;; The variable is captured. > + mapping) > + (`(car-safe (internal-get-closed-var . ,_)) > + ;; The variable is mutably captured; skip > + ;; the indirection step because the variable is > + ;; passed "by reference" to the =CE=BB-lifted function. > + (cadr mapping)) > + ((or '() `(car-safe ,(pred symbolp))) > + ;; The variable is not captured; use the (shadowed) variable valu= e. > + var)))) The docstring or comment at the beginning should mention this function is specifically for shadowed vars. Also, If mapping is of the form (car-safe SYMBOL) is `var` really the correct answer? Shouldn't it still be (cadr mapping)? > (defun cconv-convert (form env extend) > ;; This function actually rewrites the tree. > "Return FORM with all its lambdas changed so they are closed. > @@ -428,10 +444,11 @@ cconv-convert > ;; One of the lambda-lifted vars is shadowed, so add > ;; a reference to the outside binding and arrange to use > ;; that reference. > - (let ((closedsym (make-symbol (format "closed-%s" var))= )) > + (let ((var-def (cconv--lifted-arg var env)) > + (closedsym (make-symbol (format "closed-%s" var))= )) > (setq new-env (cconv--remap-llv new-env var closedsym= )) > (setq new-extend (cons closedsym (remq var new-extend= ))) > - (push `(,closedsym ,var) binders-new))) > + (push `(,closedsym ,var-def) binders-new))) Looks good. Side note: I don't understand why we `(cons closedsym`, since that `closedsym` can never appear in another binding (since it's fresh). [ Version B: ] > @@ -441,14 +442,16 @@ cconv-convert > (cconv-convert value env extend))))) >=20=20 > (when (and (eq letsym 'let*) (memq var new-extend)) > - ;; One of the lambda-lifted vars is shadowed, so add > - ;; a reference to the outside binding and arrange to use > - ;; that reference. > - (let ((var-def (cconv--lifted-arg var env)) > - (closedsym (make-symbol (format "closed-%s" var))= )) > - (setq new-env (cconv--remap-llv new-env var closedsym= )) > - (setq new-extend (cons closedsym (remq var new-extend= ))) > - (push `(,closedsym ,var-def) binders-new))) > + ;; One of the lambda-lifted vars is shadowed; if > + ;; necessary, add a reference to the outside binding > + ;; and arrange to use that reference. > + (let* ((lifted-arg (cconv--lifted-arg var env))) > + ;; This means that we may add accessors to ENV and EX= TEND > + ;; passing them off as variables, but it's close enou= gh. > + (setq new-env (cconv--remap-llv new-env var lifted-ar= g)) > + (setq new-extend (cons lifted-arg (remq var new-exten= d))) > + (when (symbolp lifted-arg) > + (push `(,lifted-arg ,var) binders-new)))) Just like I think the `(cons closedsym` is useless in the current (and in patch A), I think this `(cons lifted-arg` is not needed. I don't much like this `symbolp` test (which fundamentally seems to be trying to recover the information about which branch of the `pcase` we're coming from in `cconv--lifted-arg`). It at least deserves a comment explaining why it's doing the right thing. If we can remove this `symbolp` test recovering info about provenance of the result of `cconv--lifted-arg` then I think option B is better, but I prefer otherwise option A. Stefan