Re: Novice trouble with generator.el

[Michael Heerdegen <michael_heerdegen@web.de>]

Davin Pearson <davin.pearson@gmail.com> writes:

> The file generator.el in emacs version 25.0.50 looks like a good
> implementation of multi-threading, but the documentation is aimed at
> experts in using the system.  What I would like is a simple example of
> how to go about this multi-threading.

AFAIR Daniel planned to implement coroutines based on generator.el
later.

For now, we have the iterators defined there.  I agree that the
documentation is quite academical.

Actually, `iter-defun' is quite easy to use and understand.  It is much
like a normal defun.  If you don't call `iter-yield' or
`iter-yield-from' in the body, it is exactly defun.  Likewise
`iter-lambda'.

Let's leave `iter-yield-from' aside for a moment.

Write your `iter-defun' like normal code, like a defun that would do all
the work at once or return all elements in succession.  To return an
element and give control back to the caller, just use `iter-yield'.  To
reinvoke the generator (produce one more element), the caller should use
`iter-next' on it.  That's all.  When the body of the `iter-defun'
completes normally (i.e. terminates without calling `iter-yield') the
generator has run out of elements and stops/ your coroutine has
finished.

Here are two real life examples.  The first one is an implementation of
the sieve of Eratosthenes (don't we use this at home all the time?):

--8<---------------cut here---------------start------------->8---
(iter-defun cross-through-multiples-of (n)
  "Repeat indefinitely: Return `t' N-1 times, then return `nil' once."
  (let ((i (1- n)))
    (while t
      (if (zerop i)  (progn (setq i (1- n)) (iter-yield nil))
        (iter-yield t)
        (cl-decf i)))))
   
(iter-defun make-prime-gen ()
  "Return a generator of the prime numbers."
  (let ((n 2) (sieve '()))
    (while t
      (when (cl-every #'identity (mapcar #'iter-next sieve))
        ;; a new prime!
        (push (cross-through-multiples-of n) sieve)
        (iter-yield n))
      (cl-incf n))))
--8<---------------cut here---------------end--------------->8---
 
Both generators never finish, that's why they look like an infloop.

Test:

(setq g (make-prime-gen))

(iter-next g)

  ==> 2

(iter-next g)

  ==> 3

(setq h (make-prime-gen))

(iter-next h)

  ==> 2

(iter-next g)

  === 5


Second example: a generator that returns all files in a directory in
succession, recursively if you want:

--8<---------------cut here---------------start------------->8---
(iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)
  "Return an iterator of files in DIRECTORY.
The arguments FULL, MATCH and NOSORT are like in
`directory-files'.  Forth optional arg RECURSE non-nil means
recurse on subdirectories.  If RECURSE is a function, it should
accept one argument, a directory file name, and return non-nil
when we should recurse into that directory.  Any other non-nil
value means recurse into every readable subdirectory."
  (when (file-executable-p directory) ; i.e. DIRECTORY is readable
    (let ((files (directory-files directory full match nosort))  file)
      (while (setq file (pop files))
        (cond
         ((not (file-directory-p file))
          (iter-yield file))
         ((member (file-name-nondirectory (directory-file-name file)) '("." "..")))
         (t
          (iter-yield file)
          (when (or (and (functionp recurse)
                         (funcall recurse file))
                    recurse)
            (iter-yield-from (iterator-of-directory-files file full match nosort recurse)))))))))
--8<---------------cut here---------------end--------------->8---

`iter-yield-from' is much like `iter-yield', but instead of returning
one element, it takes an iterator as argument and returns the elements
it produces, one at a time and one after the other, until that generator
runs out, and continues afterwards.

The nomenclature of generator vs. iterator is a bit confusing in
"generator.el".  In that package, a generator is a function that, when
called, returns an iterator object.  An iterator is a function accepting
zero arguments that produces elements.

In the above examples, `cross-through-multiples-of', `make-prime-gen'
and `iterator-of-directory-files' are generators, `g' and `h' are
iterators.


HTH,

Michael.