Re: [Emacs-diffs] master f6b5db6: Add support for generators

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

>     Add support for generators
>    
>     diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
[...]
>     +	* automated/generator-tests.el: New tests

The content of the commit log is fine, but the form sucks.  Please get
rid of the diff-cruft in those commit messages.

> +  A @dfn{generator} is a function that produces a potentially-infinite
                            ^^^^^^^^
Do we want to document it as a function?  Maybe it would be better to
document it as an "object" of unspecified implementation (i.e. calling
it via funcall rather than via iter-next is unsupported).

> +@defmac iter-yield-from iterator
> +@code{iter-yield-from} yields all the values that @var{iterator}
> +produces and evaluates to the value that @var{iterator}'s generator
> +function returns normally.  While it has control, @var{iterator}
> +receives sent to the iterator using @code{iter-next}.
   ^^^^^^^^^^^^^
Some word(s) is missing here.

> +  To use a generator function, first call it normally, producing a
> +@dfn{iterator} object.

Ah, right, so that's why you defined generators as functions.  OK, now
that makes sense.  Maybe this description of the difference between
a generator and an iterator should be placed earlier, tho.

> +@defun iter-close iterator
> +If @var{iterator} is suspended inside a @code{unwind-protect} and
> +becomes unreachable, Emacs will eventually run unwind handlers after a
> +garbage collection pass.  To ensure that these handlers are run before
> +then, use @code{iter-close}.
> +@end defun

Hmm... but earlier you said:

   +@code{iter-yield} and @code{iter-yield-from} cannot appear inside
   +@code{unwind-protect} forms.

Oh, wait, I think I understand: we can use iter-yield inside the main
body of unwind-protect but not inside the unwind forms, right?  I think
this should be clarified.

To get back to iter-close, this seems to be a tricky aspect of the
implementation.  Could you give me more details, such as an example
problematic case where calling iter-close makes a difference?
I guess this has to do with those finalizers, so if you could explain
how/why these are used, I'd really appreciate it.

> +@example
> +(iter-defun my-iter (x)
> +  (iter-yield (1+ (iter-yield (1+ x))))
> +  -1 ;; Return normally
> +  )

Please don't leave a close-paren alone on its line in such example code.

> -	* vc/vc.el (vc-responsible-backend): Add autoload cooking for
> +	* vc/vc.el (vc-responsible-backend): Add autoload cookie for

Thanks ;-)

> +(defvar *cps-bindings* nil)
> +(defvar *cps-states* nil)
> +(defvar *cps-value-symbol* nil)
> +(defvar *cps-state-symbol* nil)
> +(defvar *cps-cleanup-table-symbol* nil)
> +(defvar *cps-cleanup-function* nil)
> +
> +(defvar *cps-dynamic-wrappers* '(identity)
> +  "List of transformer functions to apply to atomic forms we
> +evaluate in CPS context.")

This CL naming style is not used in Elisp.  I.e. just drop the
surrounding stars.

> +  (let* ((state (cl-gensym (format "cps-state-%s-" kind))))
                    ^^^^^^^^^
Elisp prefers make-symbol.

> +(defvar cps-disable-atomic-optimization nil

We usually use "inhibit" rather than "disable".

> +    ;; Unfortunately, because elisp lacks a mechanism for generically
> +    ;; capturing the reason for an arbitrary non-local control
> +    ;; transfer and restarting the transfer at a later point, we
> +    ;; cannot reify non-local transfers and cannot allow
> +    ;; continuation-passing code inside UNWINDFORMS.

I think it's an acceptable limitation.  Unwind-forms should be "short
and to the point".  We actually should run them with inhibit-quit (tho
we currently don't, which leads to some problems when the user hits C-g
repeatedly in panic).

> +(put 'iter-end-of-sequence 'error-conditions '(iter-end-of-sequence))
> +(put 'iter-end-of-sequence 'error-message "iteration terminated")

This should use the newish `define-error'.

> +(defmacro iter-yield-from (value)
> +  "When used inside a generator function, delegate to a sub-iterator.
> +The values that the sub-iterator yields are passed directly to
> +the caller, and values supplied to `iter-next' are sent to the
> +sub-iterator.  `iter-yield-from' evaluates to the value that the
> +sub-iterator function returns via `iter-end-of-sequence'."
> +  (let ((errsym (cl-gensym "yield-from-result"))
> +        (valsym (cl-gensym "yield-from-value")))
> +    `(let ((,valsym ,value))
> +       (unwind-protect
> +            (condition-case ,errsym
> +                (let ((vs nil))
> +                  (while t
> +                    (setf vs (iter-yield (iter-next ,valsym vs)))))
> +              (iter-end-of-sequence (cdr ,errsym)))
> +         (iter-close ,valsym)))))

[Mostly out of curiosity:] Could this be implemented as

   `(iter--blabla ,value (lambda (x) (iter-yield x)))

> +(defmacro iter-defun (name arglist &rest body)
> +  "Creates a generator NAME.
> +When called as a function, NAME returns an iterator value that
> +encapsulates the state of a computation that produces a sequence
> +of values.  Callers can retrieve each value using `iter-next'."
> +  (declare (indent defun))
> +  (cl-assert lexical-binding)
> +  `(defun ,name ,arglist
> +     ,(cps-generate-evaluator
> +       `(cl-macrolet ((iter-yield (value) `(cps-internal-yield ,value)))
> +          ,@body))))

IIUC you don't support any declarations between ARGLIST and BODY.
You could use macroexp-parse-body for that.

In cps-generate-evaluator I see you do:

> +                         (macroexpand-all form)

IIUC this will lose any local macro stored in
macroexpand-all-environment.  IOW you need to pass
macroexpand-all-environment to this macroexpand-all call.

Also, both calls to cps-generate-evaluator look identical, so I think
it'd be better to just pass `body' to cps-generate-evaluator, and then
have cps-generate-evaluator handle the local macro definition of iter-yield.

> +(eval-after-load 'elisp-mode
> +  (lambda ()
> +    (font-lock-add-keywords
> +     'emacs-lisp-mode
> +     '(("(\\(iter-defun\\)\\_>\\s *\\(\\(?:\\sw\\|\\s_\\)+\\)?"
> +        (1 font-lock-keyword-face nil t)
> +        (2 font-lock-function-name-face nil t))
> +       ("(\\(iter-next\\)\\_>"
> +        (1 font-lock-keyword-face nil t))
> +       ("(\\(iter-lambda\\)\\_>"
> +        (1 font-lock-keyword-face nil t))
> +       ("(\\(iter-yield\\)\\_>"
> +        (1 font-lock-keyword-face nil t))
> +       ("(\\(iter-yield-from\\)\\_>"
> +        (1 font-lock-keyword-face nil t))))))

Using fewer entries (at most 2) would make this more efficient
(especially since they all start with "(iter-)").


        Stefan

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 8056 bytes --]

On 03/03/2015 10:05 AM, Stefan Monnier wrote:
>>     Add support for generators
>>    
>>     diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
> [...]
>>     +	* automated/generator-tests.el: New tests
> 
> The content of the commit log is fine, but the form sucks.  Please get
> rid of the diff-cruft in those commit messages.
> 
>> +  A @dfn{generator} is a function that produces a potentially-infinite
>                             ^^^^^^^^
> Do we want to document it as a function?  Maybe it would be better to
> document it as an "object" of unspecified implementation (i.e. calling
> it via funcall rather than via iter-next is unsupported).

The iterator is an opaque object; a generator (of which an iterator is
an instance) is always a function. How can I make that clear?

>> +@defmac iter-yield-from iterator
>> +@code{iter-yield-from} yields all the values that @var{iterator}
>> +produces and evaluates to the value that @var{iterator}'s generator
>> +function returns normally.  While it has control, @var{iterator}
>> +receives sent to the iterator using @code{iter-next}.
>    ^^^^^^^^^^^^^
> Some word(s) is missing here.

Ecks. I accidentally the whole chapter.

>> +@defun iter-close iterator
>> +If @var{iterator} is suspended inside a @code{unwind-protect} and
>> +becomes unreachable, Emacs will eventually run unwind handlers after a
>> +garbage collection pass.  To ensure that these handlers are run before
>> +then, use @code{iter-close}.
>> +@end defun
> 
> Hmm... but earlier you said:
> 
>    +@code{iter-yield} and @code{iter-yield-from} cannot appear inside
>    +@code{unwind-protect} forms.
> 
> Oh, wait, I think I understand: we can use iter-yield inside the main
> body of unwind-protect but not inside the unwind forms, right?  I think
> this should be clarified.

I'll clarify it.

> To get back to iter-close, this seems to be a tricky aspect of the
> implementation.  Could you give me more details, such as an example
> problematic case where calling iter-close makes a difference?
> I guess this has to do with those finalizers, so if you could explain
> how/why these are used, I'd really appreciate it.

In conventional Emacs Lisp, we're able to use unwind-protect to clean up
resources. I'd like generator functions to be as similar to regular Lisp
code as possible, and without support for closing iterators, preferably
automatically after GC, we'll violate this user expectation. Sure, it's
probably possible to work around the absence in specific cases, but
having to very careful will lead to leaks. The Python people had to add
a very similar feature in PEP 342 for much the same reason.

>> +@example
>> +(iter-defun my-iter (x)
>> +  (iter-yield (1+ (iter-yield (1+ x))))
>> +  -1 ;; Return normally
>> +  )
> 
> Please don't leave a close-paren alone on its line in such example code.

Sure.

>> +(defvar *cps-bindings* nil)
>> +(defvar *cps-states* nil)
>> +(defvar *cps-value-symbol* nil)
>> +(defvar *cps-state-symbol* nil)
>> +(defvar *cps-cleanup-table-symbol* nil)
>> +(defvar *cps-cleanup-function* nil)
>> +
>> +(defvar *cps-dynamic-wrappers* '(identity)
>> +  "List of transformer functions to apply to atomic forms we
>> +evaluate in CPS context.")
> 
> This CL naming style is not used in Elisp.  I.e. just drop the
> surrounding stars.

Org uses that convention. But sure.

>> +  (let* ((state (cl-gensym (format "cps-state-%s-" kind))))
>                     ^^^^^^^^^
> Elisp prefers make-symbol.

cl-gensym makes it possible to actually read the code the macro
produces. A wrapper that conditionally calls make-symbol should suffice.

>> +(defvar cps-disable-atomic-optimization nil
> 
> We usually use "inhibit" rather than "disable".

Sure.

> 
>> +    ;; Unfortunately, because elisp lacks a mechanism for generically
>> +    ;; capturing the reason for an arbitrary non-local control
>> +    ;; transfer and restarting the transfer at a later point, we
>> +    ;; cannot reify non-local transfers and cannot allow
>> +    ;; continuation-passing code inside UNWINDFORMS.
> 
> I think it's an acceptable limitation.  Unwind-forms should be "short
> and to the point".  We actually should run them with inhibit-quit (tho
> we currently don't, which leads to some problems when the user hits C-g
> repeatedly in panic).

Yes, we probably should --- but if we do that, we should impose some
kind of sane time limit on unwind-protect unwindforms in order to avoid
freezing Emacs. That is, we should be able to recover from
(unwind-protect nil (while t)) without having to attach a debugger or
send a signal.

>> +(put 'iter-end-of-sequence 'error-conditions '(iter-end-of-sequence))
>> +(put 'iter-end-of-sequence 'error-message "iteration terminated")
> 
> This should use the newish `define-error'.

These conditions should have no parent, and using nil for parent in
define-error makes them have error for a parent.

>> +(defmacro iter-yield-from (value)
>> +  "When used inside a generator function, delegate to a sub-iterator.
>> +The values that the sub-iterator yields are passed directly to
>> +the caller, and values supplied to `iter-next' are sent to the
>> +sub-iterator.  `iter-yield-from' evaluates to the value that the
>> +sub-iterator function returns via `iter-end-of-sequence'."
>> +  (let ((errsym (cl-gensym "yield-from-result"))
>> +        (valsym (cl-gensym "yield-from-value")))
>> +    `(let ((,valsym ,value))
>> +       (unwind-protect
>> +            (condition-case ,errsym
>> +                (let ((vs nil))
>> +                  (while t
>> +                    (setf vs (iter-yield (iter-next ,valsym vs)))))
>> +              (iter-end-of-sequence (cdr ,errsym)))
>> +         (iter-close ,valsym)))))
> 
> [Mostly out of curiosity:] Could this be implemented as
> 
>    `(iter--blabla ,value (lambda (x) (iter-yield x)))

What do you mean?

> 
>> +(defmacro iter-defun (name arglist &rest body)
>> +  "Creates a generator NAME.
>> +When called as a function, NAME returns an iterator value that
>> +encapsulates the state of a computation that produces a sequence
>> +of values.  Callers can retrieve each value using `iter-next'."
>> +  (declare (indent defun))
>> +  (cl-assert lexical-binding)
>> +  `(defun ,name ,arglist
>> +     ,(cps-generate-evaluator
>> +       `(cl-macrolet ((iter-yield (value) `(cps-internal-yield ,value)))
>> +          ,@body))))
> 
> IIUC you don't support any declarations between ARGLIST and BODY.
> You could use macroexp-parse-body for that.

Fixed in a subsequent commit. Manually, though: maybe
macroexp-parse-body is the right thing.

> In cps-generate-evaluator I see you do:
> 
>> +                         (macroexpand-all form)
> 
> IIUC this will lose any local macro stored in
> macroexpand-all-environment.  IOW you need to pass
> macroexpand-all-environment to this macroexpand-all call.

And more generally. Thanks.

> Also, both calls to cps-generate-evaluator look identical, so I think
> it'd be better to just pass `body' to cps-generate-evaluator, and then
> have cps-generate-evaluator handle the local macro definition of iter-yield.

Sure.

>> +(eval-after-load 'elisp-mode
>> +  (lambda ()
>> +    (font-lock-add-keywords
>> +     'emacs-lisp-mode
>> +     '(("(\\(iter-defun\\)\\_>\\s *\\(\\(?:\\sw\\|\\s_\\)+\\)?"
>> +        (1 font-lock-keyword-face nil t)
>> +        (2 font-lock-function-name-face nil t))
>> +       ("(\\(iter-next\\)\\_>"
>> +        (1 font-lock-keyword-face nil t))
>> +       ("(\\(iter-lambda\\)\\_>"
>> +        (1 font-lock-keyword-face nil t))
>> +       ("(\\(iter-yield\\)\\_>"
>> +        (1 font-lock-keyword-face nil t))
>> +       ("(\\(iter-yield-from\\)\\_>"
>> +        (1 font-lock-keyword-face nil t))))))
> 
> Using fewer entries (at most 2) would make this more efficient
> (especially since they all start with "(iter-)").

Sure.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

>> Do we want to document it as a function?  Maybe it would be better to
>> document it as an "object" of unspecified implementation (i.e. calling
>> it via funcall rather than via iter-next is unsupported).

> The iterator is an opaque object; a generator (of which an iterator is
> an instance) is always a function. How can I make that clear?

As I mentioned a bit further, the explanation about the difference
between an iterator and a generator clarifies this, but comes
a bit late.

> Ecks. I accidentally the whole chapter.

[ Thanks, that was a good one.  ]

>> To get back to iter-close, this seems to be a tricky aspect of the
>> implementation.  Could you give me more details, such as an example
>> problematic case where calling iter-close makes a difference?
>> I guess this has to do with those finalizers, so if you could explain
>> how/why these are used, I'd really appreciate it.

> In conventional Emacs Lisp, we're able to use unwind-protect to clean up
> resources. I'd like generator functions to be as similar to regular Lisp
> code as possible, and without support for closing iterators, preferably
> automatically after GC, we'll violate this user expectation. Sure, it's
> probably possible to work around the absence in specific cases, but
> having to very careful will lead to leaks. The Python people had to add
> a very similar feature in PEP 342 for much the same reason.

I'm inclined to believe you that it's needed, but I'd really like to see
some good explanation for why, probably with a motivating example.
Ideally in a comment somewhere.

>>> +  (let* ((state (cl-gensym (format "cps-state-%s-" kind))))
>> ^^^^^^^^^
>> Elisp prefers make-symbol.

> cl-gensym makes it possible to actually read the code the macro
> produces. A wrapper that conditionally calls make-symbol should suffice.

A wrapper is not worth the trouble.
As for making `make-symbol' code readable, I use (setq print-gensym t)
and/or (setq print-circle t).

> Yes, we probably should --- but if we do that, we should impose some
> kind of sane time limit on unwind-protect unwindforms in order to avoid
> freezing Emacs. That is, we should be able to recover from
> (unwind-protect nil (while t)) without having to attach a debugger or
> send a signal.

We should be able to recover from an inf-loop inside inhibit-quit in any
case, yes.  Normally C-g C-g C-g does it, but it kind of defeats the
purpose in this case since users my lean on C-g without realizing that
it has a different meaning.

>> `(iter--blabla ,value (lambda (x) (iter-yield x)))
> What do you mean?

Move the unwind-protect and the loop into a separate higher-order function.

> Fixed in a subsequent commit. Manually, though: maybe
> macroexp-parse-body is the right thing.

Yes, it's the new thing recently introduced, so that it handle "all" the
special declaration-like thingies (docstrings, declare, cl-declare, you
name it).


        Stefan

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

> This CL naming style is not used in Elisp.  I.e. just drop the
> surrounding stars.

Since 24.4 we also have the convention of using a double dash on the name
of everything that's not user-facing (both variables and functions).

If it's not too much trouble, do you think that might be possible?

[-- Attachment #2: Type: text/html, Size: 371 bytes --]

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 4147 bytes --]

On 03/03/2015 11:35 AM, Stefan Monnier wrote:
>>> Do we want to document it as a function?  Maybe it would be better to
>>> document it as an "object" of unspecified implementation (i.e. calling
>>> it via funcall rather than via iter-next is unsupported).
> 
>> The iterator is an opaque object; a generator (of which an iterator is
>> an instance) is always a function. How can I make that clear?
> 
> As I mentioned a bit further, the explanation about the difference
> between an iterator and a generator clarifies this, but comes
> a bit late.

Right. I'm not sure that level of detail is appropriate for the chapter
introduction though.

>>> To get back to iter-close, this seems to be a tricky aspect of the
>>> implementation.  Could you give me more details, such as an example
>>> problematic case where calling iter-close makes a difference?
>>> I guess this has to do with those finalizers, so if you could explain
>>> how/why these are used, I'd really appreciate it.
> 
>> In conventional Emacs Lisp, we're able to use unwind-protect to clean up
>> resources. I'd like generator functions to be as similar to regular Lisp
>> code as possible, and without support for closing iterators, preferably
>> automatically after GC, we'll violate this user expectation. Sure, it's
>> probably possible to work around the absence in specific cases, but
>> having to very careful will lead to leaks. The Python people had to add
>> a very similar feature in PEP 342 for much the same reason.
> 
> I'm inclined to believe you that it's needed, but I'd really like to see
> some good explanation for why, probably with a motivating example.
> Ideally in a comment somewhere.

I don't have one yet, and that's mostly because the higher-level
facilities I want to build on top of generators don't exist yet. In
particularly, I'd like to build something that makes process filters
easier to write, that would allow you to write something like

  (let ((line (iter-yield-from (easy-filter-read-line-async mumble)))
    ;; Emacs runs while we're reading the line!
    (do-something line))

I don't want to iter-close later: if we add it, we'll change the
semantics of existing unwind-protect blocks. It feels better to do it
right from the start. I can think a few possible resources we might want
to clean up: what if we want to unload a module when it's no longer
needed? Kill a hidden buffer? Remove overlays? Kill a subprocess? Delete
a file?

>>>> +  (let* ((state (cl-gensym (format "cps-state-%s-" kind))))
>>> ^^^^^^^^^
>>> Elisp prefers make-symbol.
> 
>> cl-gensym makes it possible to actually read the code the macro
>> produces. A wrapper that conditionally calls make-symbol should suffice.
> 
> A wrapper is not worth the trouble.

I added one already. Mind leaving it there?

> As for making `make-symbol' code readable, I use (setq print-gensym t)
> and/or (setq print-circle t).
> 
>> Yes, we probably should --- but if we do that, we should impose some
>> kind of sane time limit on unwind-protect unwindforms in order to avoid
>> freezing Emacs. That is, we should be able to recover from
>> (unwind-protect nil (while t)) without having to attach a debugger or
>> send a signal.
> 
> We should be able to recover from an inf-loop inside inhibit-quit in any
> case, yes.  Normally C-g C-g C-g does it, but it kind of defeats the
> purpose in this case since users my lean on C-g without realizing that
> it has a different meaning.
> 
>>> `(iter--blabla ,value (lambda (x) (iter-yield x)))
>> What do you mean?
> 
> Move the unwind-protect and the loop into a separate higher-order function.

I'm not sure that would help. We need to keep the `iter-yield' in the
body of the calling function in order to be correct.

>> Fixed in a subsequent commit. Manually, though: maybe
>> macroexp-parse-body is the right thing.
> 
> Yes, it's the new thing recently introduced, so that it handle "all" the
> special declaration-like thingies (docstrings, declare, cl-declare, you
> name it).

Neat. Fixed. It's a shame we can't use it in `defmacro'.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 4541 bytes --]

On 03/03/2015 01:03 PM, Daniel Colascione wrote:
> On 03/03/2015 11:35 AM, Stefan Monnier wrote:
>>>> Do we want to document it as a function?  Maybe it would be better to
>>>> document it as an "object" of unspecified implementation (i.e. calling
>>>> it via funcall rather than via iter-next is unsupported).
>>
>>> The iterator is an opaque object; a generator (of which an iterator is
>>> an instance) is always a function. How can I make that clear?
>>
>> As I mentioned a bit further, the explanation about the difference
>> between an iterator and a generator clarifies this, but comes
>> a bit late.
> 
> Right. I'm not sure that level of detail is appropriate for the chapter
> introduction though.
> 
>>>> To get back to iter-close, this seems to be a tricky aspect of the
>>>> implementation.  Could you give me more details, such as an example
>>>> problematic case where calling iter-close makes a difference?
>>>> I guess this has to do with those finalizers, so if you could explain
>>>> how/why these are used, I'd really appreciate it.
>>
>>> In conventional Emacs Lisp, we're able to use unwind-protect to clean up
>>> resources. I'd like generator functions to be as similar to regular Lisp
>>> code as possible, and without support for closing iterators, preferably
>>> automatically after GC, we'll violate this user expectation. Sure, it's
>>> probably possible to work around the absence in specific cases, but
>>> having to very careful will lead to leaks. The Python people had to add
>>> a very similar feature in PEP 342 for much the same reason.
>>
>> I'm inclined to believe you that it's needed, but I'd really like to see
>> some good explanation for why, probably with a motivating example.
>> Ideally in a comment somewhere.
> 
> I don't have one yet, and that's mostly because the higher-level
> facilities I want to build on top of generators don't exist yet. In
> particularly, I'd like to build something that makes process filters
> easier to write, that would allow you to write something like
> 
>   (let ((line (iter-yield-from (easy-filter-read-line-async mumble)))
>     ;; Emacs runs while we're reading the line!
>     (do-something line))
> 
> I don't want to iter-close later: if we add it, we'll change the
> semantics of existing unwind-protect blocks. It feels better to do it
> right from the start. I can think a few possible resources we might want
> to clean up: what if we want to unload a module when it's no longer
> needed? Kill a hidden buffer? Remove overlays? Kill a subprocess? Delete
> a file?
> 
>>>>> +  (let* ((state (cl-gensym (format "cps-state-%s-" kind))))
>>>> ^^^^^^^^^
>>>> Elisp prefers make-symbol.
>>
>>> cl-gensym makes it possible to actually read the code the macro
>>> produces. A wrapper that conditionally calls make-symbol should suffice.
>>
>> A wrapper is not worth the trouble.
> 
> I added one already. Mind leaving it there?
> 
>> As for making `make-symbol' code readable, I use (setq print-gensym t)
>> and/or (setq print-circle t).
>>
>>> Yes, we probably should --- but if we do that, we should impose some
>>> kind of sane time limit on unwind-protect unwindforms in order to avoid
>>> freezing Emacs. That is, we should be able to recover from
>>> (unwind-protect nil (while t)) without having to attach a debugger or
>>> send a signal.
>>
>> We should be able to recover from an inf-loop inside inhibit-quit in any
>> case, yes.  Normally C-g C-g C-g does it, but it kind of defeats the
>> purpose in this case since users my lean on C-g without realizing that
>> it has a different meaning.
>>
>>>> `(iter--blabla ,value (lambda (x) (iter-yield x)))
>>> What do you mean?
>>
>> Move the unwind-protect and the loop into a separate higher-order function.
> 
> I'm not sure that would help. We need to keep the `iter-yield' in the
> body of the calling function in order to be correct.
> 
>>> Fixed in a subsequent commit. Manually, though: maybe
>>> macroexp-parse-body is the right thing.
>>
>> Yes, it's the new thing recently introduced, so that it handle "all" the
>> special declaration-like thingies (docstrings, declare, cl-declare, you
>> name it).
> 
> Neat. Fixed. It's a shame we can't use it in `defmacro'.

It's buggy for empty functions:

ELISP> (macroexp-parse-body '("foo" (declare indent 5)))
(("foo")
 (declare indent 5))

ELISP> (macroexp-parse-body '("foo" (declare indent 5) nil))
(("foo"
  (declare indent 5))
 nil)




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

>> As I mentioned a bit further, the explanation about the difference
>> between an iterator and a generator clarifies this, but comes
>> a bit late.
> Right. I'm not sure that level of detail is appropriate for the chapter
> introduction though.

Right.

> I don't have one yet, and that's mostly because the higher-level
> facilities I want to build on top of generators don't exist yet.

Hmm... does that mean that the current code doesn't actually use the
finalizers yet?

> I added one already. Mind leaving it there?

Your choice.

>> Move the unwind-protect and the loop into a separate higher-order function.
> I'm not sure that would help.

The question is only whether it would work.

> We need to keep the `iter-yield' in the body of the calling function
> in order to be correct.

But `iter-yield' is still (lexically) in the body of the calling
function, passed as a callback to some new higher-order function.

I guess another way to ask the same question is to ask if things like

    (mapcar (lambda (x) (iter-yield x)) list)

can be used.


        Stefan

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

> It's buggy for empty functions:
>
> ELISP> (macroexp-parse-body '("foo" (declare indent 5)))
> (("foo")
>  (declare indent 5))
>
> ELISP> (macroexp-parse-body '("foo" (declare indent 5) nil))
> (("foo"
>   (declare indent 5))
>  nil)

Looks right to me (IOW looks like a bug for a function's body to end
with a `declare').


        Stefan

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

On 03/03/2015 08:52 PM, Stefan Monnier wrote:
>> It's buggy for empty functions:
>>
>> ELISP> (macroexp-parse-body '("foo" (declare indent 5)))
>> (("foo")
>>  (declare indent 5))
>>
>> ELISP> (macroexp-parse-body '("foo" (declare indent 5) nil))
>> (("foo"
>>   (declare indent 5))
>>  nil)
> 
> Looks right to me (IOW looks like a bug for a function's body to end
> with a `declare').

Why should that be a bug?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

> Why should that be a bug?

Because the body is missing.
Don't know, maybe it's just me.  I admit having seen things like

   (defun foo ())

but it always looked like a horrible abuse to me, rather than something
I'd like to actively support.


        Stefan

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On 03/04/2015 03:02 PM, Stefan Monnier wrote:
>> Why should that be a bug?
> 
> Because the body is missing.
> Don't know, maybe it's just me.  I admit having seen things like
> 
>    (defun foo ())
> 
> but it always looked like a horrible abuse to me, rather than something
> I'd like to actively support.

It does work, though. It's analogous to (progn), (and), and (or), which
evolve to nil, nil, and t, respectively. Common Lisp also works fine
with empty function bodies. We shouldn't make empty functions work
everywhere except this one context.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: master f6b5db6: Add support for generators

[Michael Heerdegen]

Daniel Colascione <dan.colascione@gmail.com> writes:

> branch: master
> commit f6b5db6c45b773f86e203368aee9153ec8527205
> Author: Daniel Colascione <dancol@dancol.org>
> Commit: Daniel Colascione <dancol@dancol.org>
>
>     Add support for generators

Great to see that this is in master now!

I would now also like to work on including my higher-level functions if
there are no objections.

In the sense of Daniels naming conventions, this is a package about
iterators, so I think I could call it iterator.el and keep the iter-
prefix.  Would that be ok?

The package will contain a trivial constructor for ad hoc iterators a la

(defmacro iter-make-iterator (&rest body)
  `(funcall (iter-lambda () ,@body)))     ,

functions for building iterators from a list of given elements and such,
and higher order stuff (filter, append, map, reduce etc).


Thanks,

Michael.

Re: [Emacs-diffs] master f6b5db6: Add support for generators

[Stefan Monnier]

> It does work, though. It's analogous to (progn), (and), and (or), which
> evolve to nil, nil, and t, respectively. Common Lisp also works fine
> with empty function bodies. We shouldn't make empty functions work
> everywhere except this one context.

Feel free to fix macroexp-parse-body accordingly.  I don't think it's
worth my time, but I won't oppose a patch for it (of course, it still
should correctly handle the case of (defun foo () "toto")).


        Stefan

Re: master f6b5db6: Add support for generators

[Michael Heerdegen]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I would now also like to work on including my higher-level functions if
> there are no objections.

Ok, this is how the thing now looks like:

    https://github.com/michael-heerdegen/iterators.el

the file is also attached.

This is now a library of higher level stuff for the iterator objects
created by Daniel's generators that were previously included into Emacs.

What I had been named "cache" in the last version - that was a dynamic
cache for the elements returned by an iterator implemented as a delayed
list based on delayed conses - has been reimplemented and is now named
"ilist" - IteratorLIST.

These are delayed lists as well, but built from normal conses, that
(can) reference an iterator to produce "more elements" on request.  I
chose this approach because I think it suits much better the nature of
Emacs Lisp.

You can convert from/to lists and iterators, and some functions to work
with ilists are included as well.

Dunno yet how useful this stuff is, at least I need it personally.

We could put this to Gnu Elpa if Emacs Dev is interested; else, I'll put
it on Melpa.


Thanks,

Michael.



[-- Attachment #2: iterators.el --]
[-- Type: application/emacs-lisp, Size: 13827 bytes --]

Re: master f6b5db6: Add support for generators

[Stefan Monnier]

> We could put this to Gnu Elpa

Please put it in elpa.git, yes,


        Stefan

Re: master f6b5db6: Add support for generators

[Michael Heerdegen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > We could put this to Gnu Elpa
>
> Please put it in elpa.git, yes,

Sorry about my ignorance - but how do I do that?

I've cloned git://git.sv.gnu.org/emacs/elpa and have read all doc I
could find, especially README, and committed my change.

Trying to push from magit gives me:

  128 git --no-pager -c core.preloadindex=true push -v origin master:master -u
  Pushing to git+ssh://git.sv.gnu.org/srv/git/emacs/elpa
  The authenticity of host 'git.sv.gnu.org (140.186.70.72)' can't be established.
  RSA key fingerprint is [...]
  
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.

I probably don't have the "correct access rights".


Thanks in advance,

Michael.