From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: [Emacs-diffs] master f6b5db6: Add support for generators Date: Tue, 03 Mar 2015 10:20:19 -0800 Message-ID: <54F5FB63.3070203@dancol.org> References: <20150302234252.19883.48595@vcs.savannah.gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="kG8DP2A4VilgK6BtCUTaTD8DVcMjUMnXf" X-Trace: ger.gmane.org 1425406870 31968 80.91.229.3 (3 Mar 2015 18:21:10 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 3 Mar 2015 18:21:10 +0000 (UTC) To: Stefan Monnier , emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Mar 03 19:21:07 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1YSrRL-0006j3-1h for ged-emacs-devel@m.gmane.org; Tue, 03 Mar 2015 19:21:07 +0100 Original-Received: from localhost ([::1]:40300 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YSrRK-0001qa-E3 for ged-emacs-devel@m.gmane.org; Tue, 03 Mar 2015 13:21:06 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:53265) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YSrQi-0000xU-FY for emacs-devel@gnu.org; Tue, 03 Mar 2015 13:20:29 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YSrQg-0004Wn-RH for emacs-devel@gnu.org; Tue, 03 Mar 2015 13:20:28 -0500 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:60804) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YSrQg-0004W8-EN for emacs-devel@gnu.org; Tue, 03 Mar 2015 13:20:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Content-Type:In-Reply-To:References:Subject:To:MIME-Version:From:Date:Message-ID; bh=UcHRIC/maGQZJR5DUpWYJ3MqW0OncTJSIv8ViFYFdsY=; b=IbAlqnei99ySH5/eBJZ82SAYTrGWoMNmsyyr7cUMUmlhVuznD32lw6FepdF+BOdgvjPXfyK0TlNfujNW7VYmZHtb6PLml5NSy3K5fEh1ENGlti3xvO0DaGeWtf/RD+VHh8cAqYwOdq8Dc9NeOoNCNtamegGsnLsbfqg1GDQTFvfyCZch/m7Ic/E5kHLhpnic6mRjo4B+gXUuJ46YprMEz0R6WGZUFc34FVXZ4dqQp7vGMCRjbmpbpUycwqTTkEt/vbAhyk6gLbk7RWZDuDv1hndiw8oqI1+EKoNXKRTrsRkXkBBZpfDk97GC8q89EjtnVACzyAqWtNI0aJw7gUok4g==; Original-Received: from [2620:10d:c081:1101:2ab2:bdff:fe1c:db58] by dancol.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84) (envelope-from ) id 1YSrQf-0007J0-SL; Tue, 03 Mar 2015 10:20:25 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 In-Reply-To: X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:183614 Archived-At: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --kG8DP2A4VilgK6BtCUTaTD8DVcMjUMnXf Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/03/2015 10:05 AM, Stefan Monnier wrote: >> Add support for generators >> =20 >> diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog > [...] >> + * automated/generator-tests.el: New tests >=20 > The content of the commit log is fine, but the form sucks. Please get > rid of the diff-cruft in those commit messages. >=20 >> + A @dfn{generator} is a function that produces a potentially-infinit= e > ^^^^^^^^ > Do we want to document it as a function? Maybe it would be better to > document it as an "object" of unspecified implementation (i.e. calling > it via funcall rather than via iter-next is unsupported). The iterator is an opaque object; a generator (of which an iterator is an instance) is always a function. How can I make that clear? >> +@defmac iter-yield-from iterator >> +@code{iter-yield-from} yields all the values that @var{iterator} >> +produces and evaluates to the value that @var{iterator}'s generator >> +function returns normally. While it has control, @var{iterator} >> +receives sent to the iterator using @code{iter-next}. > ^^^^^^^^^^^^^ > Some word(s) is missing here. Ecks. I accidentally the whole chapter. >> +@defun iter-close iterator >> +If @var{iterator} is suspended inside a @code{unwind-protect} and >> +becomes unreachable, Emacs will eventually run unwind handlers after = a >> +garbage collection pass. To ensure that these handlers are run befor= e >> +then, use @code{iter-close}. >> +@end defun >=20 > Hmm... but earlier you said: >=20 > +@code{iter-yield} and @code{iter-yield-from} cannot appear inside > +@code{unwind-protect} forms. >=20 > Oh, wait, I think I understand: we can use iter-yield inside the main > body of unwind-protect but not inside the unwind forms, right? I think= > this should be clarified. I'll clarify it. > To get back to iter-close, this seems to be a tricky aspect of the > implementation. Could you give me more details, such as an example > problematic case where calling iter-close makes a difference? > I guess this has to do with those finalizers, so if you could explain > how/why these are used, I'd really appreciate it. In conventional Emacs Lisp, we're able to use unwind-protect to clean up resources. I'd like generator functions to be as similar to regular Lisp code as possible, and without support for closing iterators, preferably automatically after GC, we'll violate this user expectation. Sure, it's probably possible to work around the absence in specific cases, but having to very careful will lead to leaks. The Python people had to add a very similar feature in PEP 342 for much the same reason. >> +@example >> +(iter-defun my-iter (x) >> + (iter-yield (1+ (iter-yield (1+ x)))) >> + -1 ;; Return normally >> + ) >=20 > Please don't leave a close-paren alone on its line in such example code= =2E Sure. >> +(defvar *cps-bindings* nil) >> +(defvar *cps-states* nil) >> +(defvar *cps-value-symbol* nil) >> +(defvar *cps-state-symbol* nil) >> +(defvar *cps-cleanup-table-symbol* nil) >> +(defvar *cps-cleanup-function* nil) >> + >> +(defvar *cps-dynamic-wrappers* '(identity) >> + "List of transformer functions to apply to atomic forms we >> +evaluate in CPS context.") >=20 > This CL naming style is not used in Elisp. I.e. just drop the > surrounding stars. Org uses that convention. But sure. >> + (let* ((state (cl-gensym (format "cps-state-%s-" kind)))) > ^^^^^^^^^ > Elisp prefers make-symbol. cl-gensym makes it possible to actually read the code the macro produces. A wrapper that conditionally calls make-symbol should suffice. >> +(defvar cps-disable-atomic-optimization nil >=20 > We usually use "inhibit" rather than "disable". Sure. >=20 >> + ;; Unfortunately, because elisp lacks a mechanism for generically= >> + ;; capturing the reason for an arbitrary non-local control >> + ;; transfer and restarting the transfer at a later point, we >> + ;; cannot reify non-local transfers and cannot allow >> + ;; continuation-passing code inside UNWINDFORMS. >=20 > I think it's an acceptable limitation. Unwind-forms should be "short > and to the point". We actually should run them with inhibit-quit (tho > we currently don't, which leads to some problems when the user hits C-g= > repeatedly in panic). Yes, we probably should --- but if we do that, we should impose some kind of sane time limit on unwind-protect unwindforms in order to avoid freezing Emacs. That is, we should be able to recover from (unwind-protect nil (while t)) without having to attach a debugger or send a signal. >> +(put 'iter-end-of-sequence 'error-conditions '(iter-end-of-sequence))= >> +(put 'iter-end-of-sequence 'error-message "iteration terminated") >=20 > This should use the newish `define-error'. These conditions should have no parent, and using nil for parent in define-error makes them have error for a parent. >> +(defmacro iter-yield-from (value) >> + "When used inside a generator function, delegate to a sub-iterator.= >> +The values that the sub-iterator yields are passed directly to >> +the caller, and values supplied to `iter-next' are sent to the >> +sub-iterator. `iter-yield-from' evaluates to the value that the >> +sub-iterator function returns via `iter-end-of-sequence'." >> + (let ((errsym (cl-gensym "yield-from-result")) >> + (valsym (cl-gensym "yield-from-value"))) >> + `(let ((,valsym ,value)) >> + (unwind-protect >> + (condition-case ,errsym >> + (let ((vs nil)) >> + (while t >> + (setf vs (iter-yield (iter-next ,valsym vs))))) >> + (iter-end-of-sequence (cdr ,errsym))) >> + (iter-close ,valsym))))) >=20 > [Mostly out of curiosity:] Could this be implemented as >=20 > `(iter--blabla ,value (lambda (x) (iter-yield x))) What do you mean? >=20 >> +(defmacro iter-defun (name arglist &rest body) >> + "Creates a generator NAME. >> +When called as a function, NAME returns an iterator value that >> +encapsulates the state of a computation that produces a sequence >> +of values. Callers can retrieve each value using `iter-next'." >> + (declare (indent defun)) >> + (cl-assert lexical-binding) >> + `(defun ,name ,arglist >> + ,(cps-generate-evaluator >> + `(cl-macrolet ((iter-yield (value) `(cps-internal-yield ,value= ))) >> + ,@body)))) >=20 > IIUC you don't support any declarations between ARGLIST and BODY. > You could use macroexp-parse-body for that. Fixed in a subsequent commit. Manually, though: maybe macroexp-parse-body is the right thing. > In cps-generate-evaluator I see you do: >=20 >> + (macroexpand-all form) >=20 > IIUC this will lose any local macro stored in > macroexpand-all-environment. IOW you need to pass > macroexpand-all-environment to this macroexpand-all call. And more generally. Thanks. > Also, both calls to cps-generate-evaluator look identical, so I think > it'd be better to just pass `body' to cps-generate-evaluator, and then > have cps-generate-evaluator handle the local macro definition of iter-y= ield. Sure. >> +(eval-after-load 'elisp-mode >> + (lambda () >> + (font-lock-add-keywords >> + 'emacs-lisp-mode >> + '(("(\\(iter-defun\\)\\_>\\s *\\(\\(?:\\sw\\|\\s_\\)+\\)?" >> + (1 font-lock-keyword-face nil t) >> + (2 font-lock-function-name-face nil t)) >> + ("(\\(iter-next\\)\\_>" >> + (1 font-lock-keyword-face nil t)) >> + ("(\\(iter-lambda\\)\\_>" >> + (1 font-lock-keyword-face nil t)) >> + ("(\\(iter-yield\\)\\_>" >> + (1 font-lock-keyword-face nil t)) >> + ("(\\(iter-yield-from\\)\\_>" >> + (1 font-lock-keyword-face nil t)))))) >=20 > Using fewer entries (at most 2) would make this more efficient > (especially since they all start with "(iter-)"). Sure. --kG8DP2A4VilgK6BtCUTaTD8DVcMjUMnXf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJU9ftjAAoJEN4WImmbpWBl3WwP+wS6toeHYaWK+6HuFtZGYAYa mbRQib3qwcJwkcvYFuf6fQFp/fhzeWrZBx9MplsNxNbi7je9p+e3QJ4A7EfcA3BQ zg2rq0poirLmOVsEbx8Adg83I09wCK6640Wby3E1oVXIhjEr3LPtnQVfTQAGZOlh 3hG04yjPPEN1Rq8IfD3O2C8s43+fkrplyjz2DIdrprrE7VFnT9ObGbqyp6GESUVP JgQFaXtCmOvdquqVedpvqB4nTi4azUwj597M1JB2QjuI3wvcAU5VSAaEokboh46S WP+tAcWpfUFntZUw+j+ZbveRaNzVnVD3SJOTKWJ07pC//Ja3J255WJoYSpLyaJD8 N+b2jL/XJgdh6jltLsTvE2zMmOTyDv3+PzCYnLUvSHw7d05Yy3RDxK1ARCf90iJb FVVOK38iKlaWFO8UqKJY3V94JLNlhsquzqTUQwwR7n7OuOynNojfuHIcafyD72QN pruB/xIJUafTAuxohq5gKuZsuFFF5K0HrzWSOwJ/6pDwZGi5+rvE7AZt76MlZ1Th O9Vhjrx1tj2CN7y6AWcG6vbikqIQ+DoaoRS3NtrOJBaDLLBRWF+tAYjznB118G4e xTZ0wLfL5ejEj0dMnyLeYAOeCddS5NU7gv0WZEq9bPuDqjxOw59y9teTQSad6Q3Y 1o3wDNMDyzKQQ1VD/avs =f1UO -----END PGP SIGNATURE----- --kG8DP2A4VilgK6BtCUTaTD8DVcMjUMnXf--