Novice trouble with generator.el

Novice trouble with generator.el

[Davin Pearson]

The file generator.el in emacs version 25.0.50
looks like a good implementation of multi-threading,
but the documentation is aimed at experts in using
the system.  What I would like is a simple example 
of how to go about this multi-threading.

Please could one of you braniacs come up with a
simple example for dummies like myself.

Re: Novice trouble with generator.el

[Michael Heerdegen]

Davin Pearson <davin.pearson@gmail.com> writes:

> The file generator.el in emacs version 25.0.50 looks like a good
> implementation of multi-threading, but the documentation is aimed at
> experts in using the system.  What I would like is a simple example of
> how to go about this multi-threading.

AFAIR Daniel planned to implement coroutines based on generator.el
later.

For now, we have the iterators defined there.  I agree that the
documentation is quite academical.

Actually, `iter-defun' is quite easy to use and understand.  It is much
like a normal defun.  If you don't call `iter-yield' or
`iter-yield-from' in the body, it is exactly defun.  Likewise
`iter-lambda'.

Let's leave `iter-yield-from' aside for a moment.

Write your `iter-defun' like normal code, like a defun that would do all
the work at once or return all elements in succession.  To return an
element and give control back to the caller, just use `iter-yield'.  To
reinvoke the generator (produce one more element), the caller should use
`iter-next' on it.  That's all.  When the body of the `iter-defun'
completes normally (i.e. terminates without calling `iter-yield') the
generator has run out of elements and stops/ your coroutine has
finished.

Here are two real life examples.  The first one is an implementation of
the sieve of Eratosthenes (don't we use this at home all the time?):

--8<---------------cut here---------------start------------->8---
(iter-defun cross-through-multiples-of (n)
  "Repeat indefinitely: Return `t' N-1 times, then return `nil' once."
  (let ((i (1- n)))
    (while t
      (if (zerop i)  (progn (setq i (1- n)) (iter-yield nil))
        (iter-yield t)
        (cl-decf i)))))
   
(iter-defun make-prime-gen ()
  "Return a generator of the prime numbers."
  (let ((n 2) (sieve '()))
    (while t
      (when (cl-every #'identity (mapcar #'iter-next sieve))
        ;; a new prime!
        (push (cross-through-multiples-of n) sieve)
        (iter-yield n))
      (cl-incf n))))
--8<---------------cut here---------------end--------------->8---
 
Both generators never finish, that's why they look like an infloop.

Test:

(setq g (make-prime-gen))

(iter-next g)

  ==> 2

(iter-next g)

  ==> 3

(setq h (make-prime-gen))

(iter-next h)

  ==> 2

(iter-next g)

  === 5


Second example: a generator that returns all files in a directory in
succession, recursively if you want:

--8<---------------cut here---------------start------------->8---
(iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)
  "Return an iterator of files in DIRECTORY.
The arguments FULL, MATCH and NOSORT are like in
`directory-files'.  Forth optional arg RECURSE non-nil means
recurse on subdirectories.  If RECURSE is a function, it should
accept one argument, a directory file name, and return non-nil
when we should recurse into that directory.  Any other non-nil
value means recurse into every readable subdirectory."
  (when (file-executable-p directory) ; i.e. DIRECTORY is readable
    (let ((files (directory-files directory full match nosort))  file)
      (while (setq file (pop files))
        (cond
         ((not (file-directory-p file))
          (iter-yield file))
         ((member (file-name-nondirectory (directory-file-name file)) '("." "..")))
         (t
          (iter-yield file)
          (when (or (and (functionp recurse)
                         (funcall recurse file))
                    recurse)
            (iter-yield-from (iterator-of-directory-files file full match nosort recurse)))))))))
--8<---------------cut here---------------end--------------->8---

`iter-yield-from' is much like `iter-yield', but instead of returning
one element, it takes an iterator as argument and returns the elements
it produces, one at a time and one after the other, until that generator
runs out, and continues afterwards.

The nomenclature of generator vs. iterator is a bit confusing in
"generator.el".  In that package, a generator is a function that, when
called, returns an iterator object.  An iterator is a function accepting
zero arguments that produces elements.

In the above examples, `cross-through-multiples-of', `make-prime-gen'
and `iterator-of-directory-files' are generators, `g' and `h' are
iterators.


HTH,

Michael.

Re: Novice trouble with generator.el

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 05 Nov 2015 19:13:28 +0100
> 
>   (when (file-executable-p directory) ; i.e. DIRECTORY is readable

file-accessible-directory-p is better, I think.

Thanks.

Re: Novice trouble with generator.el

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> >   (when (file-executable-p directory) ; i.e. DIRECTORY is readable
>
> file-accessible-directory-p is better, I think.

I stay corrected, thanks.


Michael.

Re: Novice trouble with generator.el

[Davin Pearson]

On Friday, November 6, 2015 at 7:13:57 AM UTC+13, Michael Heerdegen wrote:
> Davin Pearson <davin dot pearson at gmail dom com> writes:
> 
> > The file generator.el in emacs version 25.0.50 looks like a good
> > implementation of multi-threading, but the documentation is aimed at
> > experts in using the system.  What I would like is a simple example of
> > how to go about this multi-threading.
> 
> AFAIR Daniel planned to implement coroutines based on generator.el
> later.
> 
> For now, we have the iterators defined there.  I agree that the
> documentation is quite academical.
> 
> Actually, `iter-defun' is quite easy to use and understand.  It is much
> like a normal defun.  If you don't call `iter-yield' or
> `iter-yield-from' in the body, it is exactly defun.  Likewise
> `iter-lambda'.
> 
> Let's leave `iter-yield-from' aside for a moment.
> 
> Write your `iter-defun' like normal code, like a defun that would do all
> the work at once or return all elements in succession.  To return an
> element and give control back to the caller, just use `iter-yield'.  To
> reinvoke the generator (produce one more element), the caller should use
> `iter-next' on it.  That's all.  When the body of the `iter-defun'
> completes normally (i.e. terminates without calling `iter-yield') the
> generator has run out of elements and stops/ your coroutine has
> finished.
> 
> Here are two real life examples.  The first one is an implementation of
> the sieve of Eratosthenes (don't we use this at home all the time?):
> 
> --8<---------------cut here---------------start------------->8---
> (iter-defun cross-through-multiples-of (n)
>   "Repeat indefinitely: Return `t' N-1 times, then return `nil' once."
>   (let ((i (1- n)))
>     (while t
>       (if (zerop i)  (progn (setq i (1- n)) (iter-yield nil))
>         (iter-yield t)
>         (cl-decf i)))))
>    
> (iter-defun make-prime-gen ()
>   "Return a generator of the prime numbers."
>   (let ((n 2) (sieve '()))
>     (while t
>       (when (cl-every #'identity (mapcar #'iter-next sieve))
>         ;; a new prime!
>         (push (cross-through-multiples-of n) sieve)
>         (iter-yield n))
>       (cl-incf n))))
> --8<---------------cut here---------------end--------------->8---
>  
> Both generators never finish, that's why they look like an infloop.
> 
> Test:
> 
> (setq g (make-prime-gen))
> 
> (iter-next g)
> 
>   ==> 2
> 
> (iter-next g)
> 
>   ==> 3
> 
> (setq h (make-prime-gen))
> 
> (iter-next h)
> 
>   ==> 2
> 
> (iter-next g)
> 
>   === 5
> 
> 
> Second example: a generator that returns all files in a directory in
> succession, recursively if you want:
> 
> --8<---------------cut here---------------start------------->8---
> (iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)
>   "Return an iterator of files in DIRECTORY.
> The arguments FULL, MATCH and NOSORT are like in
> `directory-files'.  Forth optional arg RECURSE non-nil means
> recurse on subdirectories.  If RECURSE is a function, it should
> accept one argument, a directory file name, and return non-nil
> when we should recurse into that directory.  Any other non-nil
> value means recurse into every readable subdirectory."
>   (when (file-executable-p directory) ; i.e. DIRECTORY is readable
>     (let ((files (directory-files directory full match nosort))  file)
>       (while (setq file (pop files))
>         (cond
>          ((not (file-directory-p file))
>           (iter-yield file))
>          ((member (file-name-nondirectory (directory-file-name file)) '("." "..")))
>          (t
>           (iter-yield file)
>           (when (or (and (functionp recurse)
>                          (funcall recurse file))
>                     recurse)
>             (iter-yield-from (iterator-of-directory-files file full match nosort recurse)))))))))
> --8<---------------cut here---------------end--------------->8---
> 
> `iter-yield-from' is much like `iter-yield', but instead of returning
> one element, it takes an iterator as argument and returns the elements
> it produces, one at a time and one after the other, until that generator
> runs out, and continues afterwards.
> 
> The nomenclature of generator vs. iterator is a bit confusing in
> "generator.el".  In that package, a generator is a function that, when
> called, returns an iterator object.  An iterator is a function accepting
> zero arguments that produces elements.
> 
> In the above examples, `cross-through-multiples-of', `make-prime-gen'
> and `iterator-of-directory-files' are generators, `g' and `h' are
> iterators.
> 
> 
> HTH,
> 
> Michael.

When I try to execute 

iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)

it gives me the following error:

Assertion failed: lexical-binding

What gives?  Note that I am using Emacs 24.3.1.

Do I have to execute Emacs 25.x.y?

Re: Novice trouble with generator.el

[John Mastro]

> When I try to execute 
> 
> iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)
> 
> it gives me the following error:
> 
> Assertion failed: lexical-binding
> 
> What gives?  Note that I am using Emacs 24.3.1.
> 
> Do I have to execute Emacs 25.x.y?

Put a comment like this as the very first line of the file:

;;; lexical-binding: t

Lexical binding is optional in Emacs but generator.el requires it. 

-- 
john

Re: Novice trouble with generator.el

[Davin Pearson]

On Tuesday, November 17, 2015 at 5:17:16 PM UTC+13, John Mastro wrote:
> > When I try to execute 
> > 
> > iter-defun iterator-of-directory-files (directory &optional full match nosort recurse)
> > 
> > it gives me the following error:
> > 
> > Assertion failed: lexical-binding
> > 
> > What gives?  Note that I am using Emacs 24.3.1.
> > 
> > Do I have to execute Emacs 25.x.y?
> 
> Put a comment like this as the very first line of the file:
> 
> ;;; lexical-binding: t
> 
> Lexical binding is optional in Emacs but generator.el requires it. 
> 
> -- 
> john

When I execute 

(iter-defun iterator-of-directory-files (directory &optional full match ...)

it says 

(void-function macroexp-parse-body)

I tried to install Emacs 25 using the following command:

apt-get install emacs25 

apt-get spits out the following error message:

E: Unable to locate package emacs25

How do I go about installing emacs25?

Re: Novice trouble with generator.el

[Michael Heerdegen]

Davin Pearson <davin.pearson@gmail.com> writes:

> apt-get install emacs25 
>
> apt-get spits out the following error message:
>
> E: Unable to locate package emacs25
>
> How do I go about installing emacs25?

Emacs 25 has not yet been released, so there is no such package in the
Debian repository.

You can either build it yourself locally, or find a place with such a
built.

In the past, you could install Debian packages from
http://emacs.naquadah.org/, but it seems to be abandoned.


Michael.