Generators (iterators) for Gnu Emacs

Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

Hello,

I want to contribute a package about generators to Gnu Emacs.

It is quite simple, but turned out to be surprisingly useful to have in
several situations.  It is efficient and avoids recursion - iterating
across a generator's elements results in a simple `while' loop being
executed.

Daniel Colascione suggested to me to rename the thing to "iterators.el",
since this package doesn't use continuation passing style to implement
"yield".  I guess that's reasonable.


Thanks,

Michael.



[-- Attachment #2: generator.el --]
[-- Type: application/emacs-lisp, Size: 17580 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> I want to contribute a package about generators to Gnu Emacs.

Thanks.  Looks good.  I generally prefer adding packages to GNU ELPA,
but I guess it also makes sense to add this directly to Emacs.

> Daniel Colascione suggested to me to rename the thing to "iterators.el",
> since this package doesn't use continuation passing style to implement
> "yield".  I guess that's reasonable.

I don't have any opinion on the naming.  A few comments below.


        Stefan


> ;;; Commentary:

Please state clearly which are all the basic operations one can perform on
a generator (AFAICT there's only one such operation, which is `gen-next').

It's great to see that you haven't needed anything else than `gen-next'.
I'd have expected a `gen-done-p' test to be needed/handy every once in
a while.

> (defmacro gen-make (&rest body)
>   "Return a generator that evaluates BODY to generate elements.
> For each call of `gen-next' with the the returned generator, BODY
> will be evaluated to produce an element."
>   (let ((this-element (make-symbol "this-element")))
>     `(let (,this-element)
>        (lambda ()

AFAICT this requires lexical-binding in the caller, so we might want to
signal an error if lexical-binding is nil.

>          (if (eq ,this-element 'gen-done)
>              'gen-done
>            (setq ,this-element (progn ,@body)))))))

This code breaks down if your generator happens to return the symbol
`gen-done'.  A better option is to have a (defconst gen-done <val>)
where the value is unique w.r.t `eq' (can be a string, a cons cell, an
uninterned symbol, you name it) and then to use this value rather than
hard-coding an interned symbol.

> (defun gen-next (generator)
>   (funcall generator))

A defalias would be more efficient.

>    (if (not (null elements))

Aka (if elements

> The sequence of returned elements is starting with VALUE. Any
                                                          ^^^
Add extra space (or break the line).

> successive element will be found by calling FUNCTION on the
> preceding element."
>   (gen-append
>    (gen-from-elts value)
>    (gen-make (setq value (funcall function value)))))

I guess a `gen-cons' would be more efficient than this 
(gen-append (gen-from-elts value) ...).

> (defun gen-number-range (&optional start end inc)
>   "Return a generator of a number sequence.
> START denotes the first number and defaults to 1.

Tradition in computer science is to start counting from 0.  There are
some notable exceptions (such as `point-min', which I regret), but
I strongly recommend to follow the tradition.

> The second optional argument END specifies the upper limit.  If nil,
> the returned generator is infinite.  INC is the increment used between
> numbers and defaults to 1."

Please be more precise about whether END is included or excluded.
Tradition (see again) is to exclude the upper bound (e.g. in dotimes).

> (defmacro gen-delay-expr (expression)

Have you made use of this?  I'm not sure it really fits.  I mean,
technically it works, but I'm not completely sure if pretending it's
a generator is a clever idea or not.

>   (let ((done nil) el)
>     (gen-make
>      (if done 'gen-done
>        (setq el (gen-next generator))
>        (while (and (not (eq el 'gen-done))
>                    (not (funcall predicate el)))
>          (setq el (gen-next generator)))
>        (if (eq el 'gen-done)
>            (setq done 'gen-done)
>          el)))))

Better move the `el' into the `gen-make' so it's not caught as a free
variable of the closure.  And please apply de-Morgan's law.

>     (let ((current (pop generators)) el (done nil) (skip (make-symbol "skip")))
>       (gen-delq
>        skip
>        (gen-make 
>         (if done 'gen-done
>           (setq el (gen-next current))
>           (if (eq el 'gen-done)
>               (if (null generators)
>                   (progn (setq done t) 'gen-done)
>                 (setq current (pop generators))
>                 skip)
>             el)))))))

Same here: sink `el' into the `gen-make'.

>   (nreverse (gen-reduce (lambda (x y) (cons y x)) () generator)))

Too bad the order of arguments was not swapped so we could use #'cons
instead of this lambda.

>   "Return true if PREDICATE is true for any element of GENERATOR."
            ^^^^
We call it "non-nil".  Same for `gen-every'.

>   (cl-callf or function #'identity)
>   (gen-max generator (and function (lambda (x) (- (funcall function x))))))

AFAICT, the first line makes sure `function' is not nil, so you can
remove the "(and function" on the second line.

Re: Generators (iterators) for Gnu Emacs

[Leo Liu]

On 2014-12-04 20:55 -0500, Stefan Monnier wrote:
> Thanks.  Looks good.  I generally prefer adding packages to GNU ELPA,
> but I guess it also makes sense to add this directly to Emacs.

Isn't this a iterator (or stream) instead of generator?

Having generator would be great. Daniel, how mature is your generator
package?

Leo

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On 12/04/2014 09:42 PM, Leo Liu wrote:
> On 2014-12-04 20:55 -0500, Stefan Monnier wrote:
>> Thanks.  Looks good.  I generally prefer adding packages to GNU ELPA,
>> but I guess it also makes sense to add this directly to Emacs.
> 
> Isn't this a iterator (or stream) instead of generator?
> 
> Having generator would be great. Daniel, how mature is your generator
> package?

It works well. The code generation in the CPS portions is fairly inefficient, but I don't think it matters all that much. Subroutines are compiled normally, of course, as are sub-forms that never yield.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Leo Liu]

On 2014-12-04 22:23 -0500, Daniel Colascione wrote:
> It works well. The code generation in the CPS portions is fairly
> inefficient, but I don't think it matters all that much. Subroutines
> are compiled normally, of course, as are sub-forms that never yield.

Stefan, any objection to make it part of emacs core?

Leo

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

>> It works well. The code generation in the CPS portions is fairly
>> inefficient, but I don't think it matters all that much. Subroutines
>> are compiled normally, of course, as are sub-forms that never yield.
> Stefan, any objection to make it part of emacs core?

Are we still talking about Micheal's code?


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Thierry Volpiatto]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hello,
>
> I want to contribute a package about generators to Gnu Emacs.
>
> It is quite simple, but turned out to be surprisingly useful to have in
> several situations.  It is efficient and avoids recursion - iterating
> across a generator's elements results in a simple `while' loop being
> executed.
>
> Daniel Colascione suggested to me to rename the thing to "iterators.el",

We already have iterator.el.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Generators (iterators) for Gnu Emacs

[Leo Liu]

On 2014-12-04 23:19 -0500, Stefan Monnier wrote:
> Are we still talking about Micheal's code?
>
>
>         Stefan

No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
you have time to critique the code and make it an integral part of
elisp? Thanks ;)

Leo

Re: Generators (iterators) for Gnu Emacs

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (defmacro gen-make (&rest body)
>>   "Return a generator that evaluates BODY to generate elements.
>> For each call of `gen-next' with the the returned generator, BODY
>> will be evaluated to produce an element."
>>   (let ((this-element (make-symbol "this-element")))
>>     `(let (,this-element)
>>        (lambda ()
>
> AFAICT this requires lexical-binding in the caller, so we might want to
> signal an error if lexical-binding is nil.

Uh what?  The whole point of the make-symbol abomination is to get a
symbol uniquely used by only one instance of the lambda function.  This
is _exactly_ in order to _not_ require lexical binding.

>>          (if (eq ,this-element 'gen-done)
>>              'gen-done
>>            (setq ,this-element (progn ,@body)))))))

With lexical binding, this would look something like

(defmacro gen-mane (&rest body)
  `(let (this-element)
    (lambda ()
      (if (eq this-element 'gen-done)
          'gen-done
          (setq this-element (progn ,@body))))))

And one would be tempted to see whether this really needs to be a macro.

-- 
David Kastrup

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> No.  Daniel's package https://github.com/dcolascione/elisp-generators.
> Do you have time to critique the code and make it an integral part of
> elisp? Thanks ;)

I haven't had time to look at it in detail, but it looks nice.
Things I noticed:
- Drop the use of (require 'cl).
- Use `declare' for indentation and edebug specs.
- Most importantly: clean up the namespace use.  `yield' is probably OK
  (tho the global definition should probably not override a previous
  definition) but `next' needs to turn into something like `gen-next',
  `do-iterator' should probably turn into `gen-do', cl-loop's
  `iterating' should probably turn into something that includes
  "generator" in its name, `lambda-generator' should maybe be renamed to
  `gen-lambda', ...

Once that's fixed, I'd be happy to see it included in Emacs.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
> you have time to critique the code and make it an integral part of
> elisp? Thanks ;)

Oh, one more thing: is it technically important to sign the end by
signaling a `stop-iteration', rather than by returning a special value?
I think I'd prefer returning a special value (`condition-case' tends to
be on the costly side).


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> We already have iterator.el.

And it seems like it does pretty much the same thing as Michael's
library (at least `iter-next' looks eerily similar to `gen-next').

Is there a chance you two could merge those two libraries (I think
iterator.el is a good name, since generator.el is already taken by
Daniel's library which does something related yet different)?
Then we could install the result into Emacs?


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Uh what?  The whole point of the make-symbol abomination is to get a
> symbol uniquely used by only one instance of the lambda function.
> This is _exactly_ in order to _not_ require lexical binding.

AFAICT this make-symbol is there to avoid name capture.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

On 12/05/2014 09:48 AM, Stefan Monnier wrote:
>> No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
>> you have time to critique the code and make it an integral part of
>> elisp? Thanks ;)
> 
> Oh, one more thing: is it technically important to sign the end by
> signaling a `stop-iteration', rather than by returning a special value?
> I think I'd prefer returning a special value (`condition-case' tends to
> be on the costly side).

I'd really rather signal to stop iteration. That way, it's harder to ignore end-of-iteration and loop forever and it remains possible to iterate over *any* value. condition-case is setjmp. I'm not *that* concerned by performance. The only other reasonable alternative is for iterators to yield a cons (terminal-p . value), and consing is going to be much worse than condition-case.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Uh what?  The whole point of the make-symbol abomination is to get a
> > symbol uniquely used by only one instance of the lambda function.
> > This is _exactly_ in order to _not_ require lexical binding.
>
> AFAICT this make-symbol is there to avoid name capture.

Yes, that had been the intention. To make it work with dynamic callers,
I would have to defvar the uninterned symbol. Dunno if it's worth it.

Michael.

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Is there a chance you two could merge those two libraries (I think
> iterator.el is a good name, since generator.el is already taken by
> Daniel's library which does something related yet different)?
> Then we could install the result into Emacs?

I think we could do so, though, apart from the definition, our packages
seem quite disjoint.

Anyway, it would be good if Daniel's generators would then be valid
iterators in terms of the new iterator.el.

Michael.

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>  A few comments below.

Thanks.  Still using the term "generator" to avoid confusion, though I'm
going to rename the thing to "iterator".

> It's great to see that you haven't needed anything else than `gen-next'.
> I'd have expected a `gen-done-p' test to be needed/handy every once in
> a while.

Dunno if that would be useful, I never missed it, but it should be
trivial to introduce.

> I guess a `gen-cons' would be more efficient than this 
> (gen-append (gen-from-elts value) ...).

I'm not sure implementing `gen-cons' is a good idea.  Generators are not
built from conses - but having `gen-cons' would allow to build generator
pseudo lists.  This package is about introducing a different concept.
Plus, using `gen-cons' as a building block of generators would create
deeply nested lambdas that would soon hit `max-lisp-eval-depth' when
requesting elements.  Avoiding recursion, which Emacs is not good at, is
crucial allover in this package.

Pushing to generators should not be needed repeatedly - if you think you
want this, use lists.  In the above case, it is just coincidence that
the first generator has only one element. And I don't think efficiency
is an issue here.  If you think it is I will rewrite that code, but
avoiding a `gen-cons'.

> > (defmacro gen-delay-expr (expression)
>
> Have you made use of this?  I'm not sure it really fits.  I mean,
> technically it works, but I'm not completely sure if pretending it's
> a generator is a clever idea or not.

I defined it because I need it in `gen-cache', and because it wasn't
existing yet in Emacs.  Do you think delay-expression should be defined
somewhere else in Emacs?  Then I would be happy to use it for
`gen-cache'.


Thanks,

Michael.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> I'd really rather signal to stop iteration. That way, it's harder to ignore
> end-of-iteration and loop forever and it remains possible to iterate over
> *any* value. condition-case is setjmp. I'm not *that* concerned by
> performance. The only other reasonable alternative is for iterators to
> yield a cons (terminal-p . value), and consing is going to be much worse
> than condition-case.

The alternative I was thinking was to return a unique value, as in:

   (defvar gen-eog (make-symbol "¡end-of-generator!"))
   (defun gen-eog-p (x) (eq x gen-oeg))

In terms of efficiency it should be fine (contrary to consing, indeed).


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Yes, that had been the intention. To make it work with dynamic callers,
> I would have to defvar the uninterned symbol.

Not only that, you'd have to build the lambda at run-time (i.e. make
the closures "by hand").

> Dunno if it's worth it.

It's not.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Yes, that had been the intention. To make it work with dynamic callers,
> > I would have to defvar the uninterned symbol.
>
> Not only that, you'd have to build the lambda at run-time (i.e. make
> the closures "by hand").

Would this version fix the issue?

--8<---------------cut here---------------start------------->8---
(defmacro gen-make (&rest body)
  `(gen-make-1 (lambda () ,@body)))

(defun gen-make-1 (fun)
  (let (this-element)
    (lambda ()
      (if (eq this-element 'gen-done)
          'gen-done
        (setq this-element (funcall fun))))))
--8<---------------cut here---------------end--------------->8---

Though, this `gen-make' is still inferior without lexical binding, but
at least should avoid the error.

Michael.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> I think we could do so, though, apart from the definition, our packages
> seem quite disjoint.

I prefer to look at it as "complementary".

> Anyway, it would be good if Daniel's generators would then be valid
> iterators in terms of the new iterator.el.

Indeed, that would be nice.  Can you try and work that out?


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

>> I guess a `gen-cons' would be more efficient than this 
>> (gen-append (gen-from-elts value) ...).
> I'm not sure implementing `gen-cons' is a good idea.

I only meant it as an optimization of (gen-append (gen-from-elts value) ...).
This form appears twice in your file.

>> > (defmacro gen-delay-expr (expression)
>> Have you made use of this?  I'm not sure it really fits.  I mean,
>> technically it works, but I'm not completely sure if pretending it's
>> a generator is a clever idea or not.
> I defined it because I need it in `gen-cache', and because it wasn't
> existing yet in Emacs.  Do you think delay-expression should be defined
> somewhere else in Emacs?  Then I would be happy to use it for
> `gen-cache'.

Not sure.  There's some overlap with url-future.el (which only has the
"url-" prefix because we didn't want to grab the "future-" prefix
without first getting more experience with the corresponding
abstraction).


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On 12/05/2014 01:32 PM, Stefan Monnier wrote:
>> I'd really rather signal to stop iteration. That way, it's harder to ignore
>> end-of-iteration and loop forever and it remains possible to iterate over
>> *any* value. condition-case is setjmp. I'm not *that* concerned by
>> performance. The only other reasonable alternative is for iterators to
>> yield a cons (terminal-p . value), and consing is going to be much worse
>> than condition-case.
> 
> The alternative I was thinking was to return a unique value, as in:
> 
>    (defvar gen-eog (make-symbol "¡end-of-generator!"))
>    (defun gen-eog-p (x) (eq x gen-oeg))
> 

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?
	


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

On 12/05/2014 02:16 PM, Stefan Monnier wrote:
>> I think we could do so, though, apart from the definition, our packages
>> seem quite disjoint.
> 
> I prefer to look at it as "complementary".
> 
>> Anyway, it would be good if Daniel's generators would then be valid
>> iterators in terms of the new iterator.el.
> 
> Indeed, that would be nice.  Can you try and work that out?

Assuming both packages are suitable for inclusion into core, we can definitely make them work together. I'd still very much like to use nonlocal control flow instead of a sentinel for enumeration termination, however.  

That aside, I'm not sure I agree with your namespace preferences. `yield' is fundamentally a lexically-scoped macro. Making it defer to some prior definition makes no sense, since any code that can see this macro is already written with generators in mind.

I'd also strongly prefer using `next' as the iter-get function. If we're going to make the iteration protocol a core part of the environment, it deserves a prominent place in the global namespace. Using `funcall' is inadequate: first, it doesn't signal iteration in calling code, which can lead to confusion, and second, `funcall' doesn't work if we want to extent `next' to support things like lists, for which `next' would be equivalent to `pop'. AFAICT, nothing is already squatting on `next', so binding the symbol's function slot would be harmless for a compatibility perspective.

I'd also prefer using `iterating' as the cl-loop keyword. Because the keyword is only active in the lexical scope of a cl-loop construct, there's an even lower risk of collision, and the word "iterating" is exactly the right thing in the cl-loop language: (cl-loop for x iterating y collect x) reads very well.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I only meant it as an optimization of (gen-append (gen-from-elts
> value) ...).  This form appears twice in your file.

Ok, I then can do it using some non-advertised helper function.

> Not sure.  There's some overlap with url-future.el (which only has the
> "url-" prefix because we didn't want to grab the "future-" prefix
> without first getting more experience with the corresponding
> abstraction).

I would never have found that myself. But this looks good, it's like my
gen-delay plus better error handling.  Hope it looses the url- prefix
some day.


Michael.

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Daniel Colascione <dancol@dancol.org> writes:

> Assuming both packages are suitable for inclusion into core, we can
> definitely make them work together.

Thinking a bit about it, I guess my (only) constructor:

(defmacro gen-make (&rest body)
  "Return a generator that evaluates BODY to generate elements.
  (let ((this-element (make-symbol "this-element")))
    `(let (,this-element)
       (lambda ()
         (if (eq ,this-element 'gen-done)
             'gen-done
           (setq ,this-element (progn ,@body)))))))

is just a special case of Daniel's `lambda-generator':

(defmacro gen-make (expr)
  `(lambda-generator ()
     (while t (yield ,expr))))

modulo (at least currently) the different kind of termination (right?).

So, all of my stuff can be expressed in terms of Daniels generators.
Which means that it would be a suitable extension to Daniels package,
defining some special generators, the transducers, and the cache, for
his more general kind of generators.  It could then be named
"generator-x" or so.

Michael.

Re: Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> So, all of my stuff can be expressed in terms of Daniels generators.

> It could then be named "generator-x" or so.

The same is true for Thierry's code, I think, it could extend
Daniel's library too.


Michael.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Would this version fix the issue?

No, with dynamic binding this just doesn't work.  And we don't care,
because people can use lexical-binding just fine.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Assuming both packages are suitable for inclusion into core, we can
> definitely make them work together.

That's the idea, yes.

> I'd still very much like to use nonlocal control flow instead of
> a sentinel for enumeration termination, however.

I'll let you guys decide together.  I stated my preference, but I have
better things to worry about.

> That aside, I'm not sure I agree with your namespace preferences. `yield' is
> fundamentally a lexically-scoped macro.

The lexically scoped `yield' is OK.  But the global (defmacro yield ...)
is not (tho, I'm OK with keeping it if wrapped in (unless (fboundp
'yield) ...)).

> I'd also strongly prefer using `next' as the iter-get function.

I know, but this one is out of the question.  All globally-visible names
need to be namespace-clean.  So it can be `gen-next'.

> I'd also prefer using `iterating' as the cl-loop keyword.

I know, but being globally visible, this is also out of the question: it
requires something with a "gen-" or at least some "generator" in its
name somewhere.

> Because the keyword is only active in the lexical scope of a cl-loop
> construct, there's an even lower risk of collision, and the word
> "iterating" is exactly the right thing in the cl-loop language:
> (cl-loop for x iterating y collect x) reads very well.

But it doesn't say we're iterating on your particular kind of iteratable
construct.
[ BTW: regardless of namespace cleanliness, I don't find "iterating"
  above to be a good choice, because "x" doesn't "iterate y".
  The "cl-loop" doesn't do anything else than iterate, so "iterating" is
  almost necessarily a misnomer in there.  From an English language
  point of view, I'd expect something more like "for x generated-by y"
  or "for x in-generator y" or "for x in y".  ]


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On 12/05/2014 11:09 PM, Stefan Monnier wrote:
>> That aside, I'm not sure I agree with your namespace preferences. `yield' is
>> fundamentally a lexically-scoped macro.
> 
> The lexically scoped `yield' is OK.  But the global (defmacro yield ...)
> is not (tho, I'm OK with keeping it if wrapped in (unless (fboundp
> 'yield) ...)).
> 
>> I'd also strongly prefer using `next' as the iter-get function.
> 
> I know, but this one is out of the question.  All globally-visible names
> need to be namespace-clean.  So it can be `gen-next'.

It *is* "namespace-clean", as I explained above. It's ludicrous to
expect that all future additions to elisp include horribly long,
hyphenated names even when they're describing fundamental features of
the environment. It's `car', goddammit it, not `seq-list-car'.

The point is to minimize the possibility of collision with useful user
code, not to religiously ensure that all symbols match ^[a-z]+-[a-z-]+$


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Stephen J. Turnbull]

Daniel Colascione writes:
 > On 12/05/2014 01:32 PM, Stefan Monnier wrote:

 > > The alternative I was thinking was to return a unique value, as in:
 > > 
 > >    (defvar gen-eog (make-symbol "¡end-of-generator!"))
 > >    (defun gen-eog-p (x) (eq x gen-oeg))
 > > 
 > 
 > Right. That's the approach I don't like: now there's a value you
 > can't enumerate. What if you wanted to enumerate, say, the values
 > of all symbols?

The existence of `make-symbol' already means you can't do that.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> the environment. It's `car', goddammit it, not `seq-list-car'.

Yup.  `car' sucks as well, but it's too late to fix it.
The prefix also helps discoverability, BTW.

        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

On 12/06/2014 05:24 PM, Stefan Monnier wrote:
>> the environment. It's `car', goddammit it, not `seq-list-car'.
> 
> Yup.  `car' sucks as well, but it's too late to fix it.

No, `car' is fine: it's short and to the point. `next' would be too,
except that you insist that additions to the Emacs core look like a bowl
of corned beef hash, with chunks of `iter-'-this and `seq-' that
embedded all over the code for no good reason.

Prefixes are great for making sure out-of-tree packages don't conflict
with each other. For Emacs core functionality, requiring prefixes adds
friction and subtracts elegance, and because we manage all the names in
the core *anyway*, there's no risk of a conflict.

Any user code using a name like `next' for itself deserves to break, and
at least that pain is short-lived, while requiring some uselessly long
prefix for internal symbols makes life a little bit more difficult for
everyone forever.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> you insist that additions to the Emacs core look like a bowl of corned
> beef hash, with chunks of `iter-'-this and `seq-' that embedded all
> over the code for no good reason.

Indeed I do.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On 12/06/2014 10:12 PM, Stefan Monnier wrote:
>> you insist that additions to the Emacs core look like a bowl of corned
>> beef hash, with chunks of `iter-'-this and `seq-' that embedded all
>> over the code for no good reason.
> 
> Indeed I do.

And you're really not interested in a discussion of your motives or the
circumstance under which you'd make an exception, are you? That's
supposed to motivate more contributions to core? Another recent thread
mentions the problem of too little development happening in-tree. I
wonder why that might be the case.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> the environment. It's `car', goddammit it, not `seq-list-car'.
>
> Yup.  `car' sucks as well, but it's too late to fix it.
> The prefix also helps discoverability, BTW.

What was it again?  "contents of address register" or something as
opposed to "contents of decrement register".  Rather arbitrary terms for
first element and the tail of a list.  If anything, I'd have expected
the tail to be in an address register.

But that's heritage.  Not even Scheme dared touching it.

-- 
David Kastrup

Re: Generators (iterators) for Gnu Emacs

[Stephen J. Turnbull]

David Kastrup writes:

 > What was it again?  "contents of address register" or something as
 > opposed to "contents of decrement register".  Rather arbitrary terms for
 > first element and the tail of a list.  If anything, I'd have expected
 > the tail to be in an address register.

Bigendian word packing, IIRC.

Re: Generators (iterators) for Gnu Emacs

[Ted Zlatanov]

On Sat, 06 Dec 2014 23:51:53 -0800 Daniel Colascione <dancol@dancol.org> wrote: 

DC> On 12/06/2014 10:12 PM, Stefan Monnier wrote:
>>> you insist that additions to the Emacs core look like a bowl of corned
>>> beef hash, with chunks of `iter-'-this and `seq-' that embedded all
>>> over the code for no good reason.
>> 
>> Indeed I do.

DC> And you're really not interested in a discussion of your motives or the
DC> circumstance under which you'd make an exception, are you? That's
DC> supposed to motivate more contributions to core? Another recent thread
DC> mentions the problem of too little development happening in-tree. I
DC> wonder why that might be the case.

I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp.  The closest thing I've seen is
https://github.com/Bruce-Connor/names which can be installed from the
GNU ELPA.  Perhaps that's an actual problem we keep ignoring because
it's easy to hack around it "just that once."

Ted

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> And you're really not interested in a discussion of your motives or the
> circumstance under which you'd make an exception, are you?

I think we've already discussed it at length in the cl-lib era.
It's largely an aesthetic issue: you want to save those 4 "gen-"
characters, whereas I want to keep the namespace clean.  As mentioned
I also want to keep things discoverable and modular, and given the lack
of real modules, using a "gen-" prefix is the way to make things
discoverable and modular.  Note that when I code in languages with good
module systems, I still use "prefixes", tho instead of "gen-" it would
probably be shortened to "G." or something like that.

As can be seen with the seq.el library, I want to move towards *more*
use of prefixes even for "core" functionality.  This has been the case
for many many years.  Even back in Emacs-21 times, my newcomment.el did
that, taking non-namespace-clean code from simple.el (or maybe it was
subr.el, can't remember) and making it use "comment-".

Over the years, my belief in the importance of using such prefixes has
only strengthened.  I know many people disagree, but at this point
there's really not much room for negotiation.
We should simply agree to disagree.


        Stefan


PS: In the case of "next", we could solve the namespace issue without
modules if Elisp had an equivalent to "foo.next".  But the only OO-style
system we have uses methods which have global scope, so we'd need
another object system with a different method-call syntax like maybe
(foo :next).

Re: Generators (iterators) for Gnu Emacs

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> PS: In the case of "next", we could solve the namespace issue without
> modules if Elisp had an equivalent to "foo.next".  But the only OO-style
> system we have uses methods which have global scope, so we'd need
> another object system with a different method-call syntax like maybe
> (foo :next).

So agree to add some more core things to make the problem easier.

A with-symbol for example.


Nic

Namespace macros (was: Generators (iterators) for Gnu Emacs)

[Helmut Eller]

On Sun, Dec 07 2014, Ted Zlatanov wrote:

> I don't know Stefan's motives, but would love to see actual namespaces
> in Emacs Lisp.  The closest thing I've seen is
> https://github.com/Bruce-Connor/names which can be installed from the
> GNU ELPA.  Perhaps that's an actual problem we keep ignoring because
> it's easy to hack around it "just that once."

Thanks for the pointer.

I also wrote a namespace macro: https://github.com/ellerh/namespace.el
Another one I know is: https://github.com/chrisbarrett/elisp-namespaces

The experience with my own namespace macro was that it works fairly well
when writing a library from scratch, in particular when the code
consists mostly of functions and almost no quoted symbols.  If it works
well, then the code looks a lot neater with namespaces.

But it was also rather useless when I tried it to use it with the SLIME
codebase.  The existing code uses prefixes in inconsistent ways and
quoted symbols are needed surprisingly often.

This may also be a bit of a chicken-and-egg problem: if the existing
code can't be used with those namespace macros than nobody will use
namespaces, but as soon as some important libraries can be used with
namespaces it pays of to use namespaces and we get more usable
libraries.

Helmut

RE: Generators (iterators) for Gnu Emacs

[Drew Adams]

> would love to see actual namespaces in Emacs Lisp.

+1

Prefixes are a lousy workaround.  Something like Common Lisp
packages would be good.  Explicit import/export, allowing
reference without the package prefix, etc.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> So agree to add some more core things to make the problem easier.

I think I'd be willing to add Objects, indeed.  At least the
completion-tables scream "make me an object".

> A with-symbol for example.

I don't know what `with-symbol' you're referring to.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So agree to add some more core things to make the problem easier.
>
> I think I'd be willing to add Objects, indeed.  At least the
> completion-tables scream "make me an object".
>
>> A with-symbol for example.
>
> I don't know what `with-symbol' you're referring to.

It's just a simple symbol-let, we discussed it before here I think. An
alternative to a name system is to define a namespace with a macro:

(defmacro my-package (&rest body)
  (symbol-macrolet ((foo my-package--foo)
                     (bar my-package--bar))
       ,@body))

But that isn't exactly a friendly pattern is it?


Nic

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On 12/07/2014 01:17 PM, Stefan Monnier wrote:
>> So agree to add some more core things to make the problem easier.
> 
> I think I'd be willing to add Objects, indeed.  At least the
> completion-tables scream "make me an object".

We *have* objects. Nothing is stopping you from using EIEIO for
completion tables. But I don't think you're talking about polymorphism
and encapsulation here: ITYM that we can solve the global namespace
problem by attacking the "global" part instead of the "namespace" part.
If we have your "Objects", many functions would be instead methods (or
messages or whatever terminology you want to choose) that have meaning
only in relation to one or more of the values acted upon by these
functions, without global names of their own.

That's a terrible idea. It makes it much harder to reason about the
correct operation of programs, since now it makes it virtually
impossible to implement find-definition functionality without lots of
type inference. There's also the runtime overhead of dynamic dispatch.
Using "Objects" doesn't even solve the problem it sets out to solve,
since you still need elaborately-mangled global names for the types of
the objects themselves.

Common Lisp solved this problem 20 years ago with namespaces. We should
just implement CL namespaces instead of trying to shoehorn Smalltalk
into the language. Would you accept a CL packages implementation?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: Generators (iterators) for Gnu Emacs

[Drew Adams]

> Common Lisp solved this problem 20 years ago with namespaces. We
> should just implement CL namespaces instead of trying to shoehorn
> Smalltalk into the language. Would you accept a CL packages
> implementation?

+1

(And it was 30 years ago.)

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> That's a terrible idea. It makes it much harder to reason about the
> correct operation of programs, since now it makes it virtually
> impossible to implement find-definition functionality without lots of
> type inference.

That's indeed the main problem, and I agree it's a serious one.

> There's also the runtime overhead of dynamic dispatch.

That's not relevant: one way or another there is dynamic dispatch,
regardless what system we use.

> Common Lisp solved this problem 20 years ago with namespaces. We should
> just implement CL namespaces instead of trying to shoehorn Smalltalk
> into the language. Would you accept a CL packages implementation?

Last time this was discussed (a year ago, maybe?), we neded turning it
down, mostly because `M-x grep RET' can't be used for find-definition
and find-uses.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

On 12/07/2014 03:31 PM, Stefan Monnier wrote:
>> That's a terrible idea. It makes it much harder to reason about the
>> correct operation of programs, since now it makes it virtually
>> impossible to implement find-definition functionality without lots of
>> type inference.
> 
> That's indeed the main problem, and I agree it's a serious one.
> 
>> There's also the runtime overhead of dynamic dispatch.
> 
> That's not relevant: one way or another there is dynamic dispatch,
> regardless what system we use.

That's not true: if we send messages to objects, we can only know the
code to execute after examining the object. If we invoke functions
directly, we know *statically* what code runs where.

>> Common Lisp solved this problem 20 years ago with namespaces. We should
>> just implement CL namespaces instead of trying to shoehorn Smalltalk
>> into the language. Would you accept a CL packages implementation?
> 
> Last time this was discussed (a year ago, maybe?), we neded turning it
> down, mostly because `M-x grep RET' can't be used for find-definition
> and find-uses.

*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage. At least namespaces make it relatively
symbol to determine the fully-resolved symbol from lexical context
without requiring a full type inference pass.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Namespace macros (was: Generators (iterators) for Gnu Emacs)

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On 7 Dec 2014 15:40, "Helmut Eller" <eller.helmut@gmail.com> wrote:
>
> On Sun, Dec 07 2014, Ted Zlatanov wrote:
>
> > I don't know Stefan's motives, but would love to see actual namespaces
> > in Emacs Lisp.  The closest thing I've seen is
> > https://github.com/Bruce-Connor/names which can be installed from the
> > GNU ELPA.  Perhaps that's an actual problem we keep ignoring because
> > it's easy to hack around it "just that once."
>
> Thanks for the pointer.
>
> I also wrote a namespace macro: https://github.com/ellerh/namespace.el
> Another one I know is: https://github.com/chrisbarrett/elisp-namespaces
>
> The experience with my own namespace macro was that it works fairly well
> when writing a library from scratch, in particular when the code
> consists mostly of functions and almost no quoted symbols.  If it works
> well, then the code looks a lot neater with namespaces.
>
> But it was also rather useless when I tried it to use it with the SLIME
> codebase.  The existing code uses prefixes in inconsistent ways and
> quoted symbols are needed surprisingly often.

Names.el partially solves that by namespacing sharp-quoted symbols as
function names (you have to use the sharp-quote on function names for this
to work, but that's good practice anyway). This catches the majority of the
uses of quotes for me.

Quoted variable names are impossible to identify with certainty, so Names
doesn't try to namespace them by default. But but it does offer the option
to namespace regular quoted symbols as variable names.

[-- Attachment #2: Type: text/html, Size: 2050 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Common Lisp solved this problem 20 years ago with namespaces. We should
>> just implement CL namespaces instead of trying to shoehorn Smalltalk
>> into the language. Would you accept a CL packages implementation?
>
> Last time this was discussed (a year ago, maybe?), we neded turning it
> down, mostly because `M-x grep RET' can't be used for find-definition
> and find-uses.

Could we revisit this discussion, maybe? Having a Lisp VM at the core of
Emacs means we don't need to use grep for find-definition.

And find-uses can be implemented in a lisp-only namespace-aware
searching function.

Clojure (a rather popular Lisp) uses what sounds to be a similar
packages system, and the popular editing environments implement both
above operations without considerable problems.

Re: Generators (iterators) for Gnu Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't know Stefan's motives, but would love to see actual namespaces
  > in Emacs Lisp.

The idea sounds nice in theory, but in practice multiple name spaces
do not fit into Lisp very well.  Common Lisp packages are an unclean
kludge; this was clear to me when I implemented them in the 1980s in
the Lisp Machine.  It is impossible to use them in the way one would
wish to use them.

In practice, you have to write the package prefix whenever you refer
to a symbol that has one.  It might as well be part of the symbol name
itself.  Thus, packages complicate the language definition while
providing no benefit.

So in GNU Emacs I decided to make them part of the symbol name itself
and not have packages.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

On 12/07/2014 04:25 PM, Richard Stallman wrote:
>   > I don't know Stefan's motives, but would love to see actual namespaces
>   > in Emacs Lisp.
> 
> The idea sounds nice in theory, but in practice multiple name spaces
> do not fit into Lisp very well.  Common Lisp packages are an unclean
> kludge; this was clear to me when I implemented them in the 1980s in
> the Lisp Machine.  It is impossible to use them in the way one would
> wish to use them.

The entire Common Lisp ecosystem is a counterexample so your rather
sweeping declaration. In practice, namespaces are rather useful despite
the need to *sometimes* fully *some* symbols. It's still better than
having to fully specify *every* symbol on *every* use.

> In practice, you have to write the package prefix whenever you refer
> to a symbol that has one.  It might as well be part of the symbol name
> itself.  Thus, packages complicate the language definition while
> providing no benefit.

There's a lot of Lisp code out there that uses packages in the way they
were meant to be used, and your personal distaste for the mechanism
shouldn't keep namespaces out of elisp.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> *Any* solution that frees developers from spelling all symbols in full
> will have that disadvantage.

That's right.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

On 12/07/2014 06:23 PM, Stefan Monnier wrote:
>> *Any* solution that frees developers from spelling all symbols in full
>> will have that disadvantage.
> 
> That's right.

So what? There are definitely offsetting benefits. Or are you saying M-x
grep is our top priority now?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Could we revisit this discussion, maybe? Having a Lisp VM at the core of
> Emacs means we don't need to use grep for find-definition.

I'm sure it can be made to work, with enough work.  But currently

   M-x grep RET grep foo-bar {emacs,elpa}/**/*.el RET

works pretty well, for *very* little effort.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Dmitry Gutov]

On 12/08/2014 04:25 AM, Stefan Monnier wrote:
>> Could we revisit this discussion, maybe? Having a Lisp VM at the core of
>> Emacs means we don't need to use grep for find-definition.
>
> I'm sure it can be made to work, with enough work.  But currently
>
>     M-x grep RET grep foo-bar {emacs,elpa}/**/*.el RET
>
> works pretty well, for *very* little effort.

Sure. In no way I'm trying to say that having to implement the above 
would be much of a benefit by itself.

It's a cost, for having a modern-ish import system.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

>>> *Any* solution that frees developers from spelling all symbols in full
>>> will have that disadvantage.
>> That's right.
> So what?  There are definitely offsetting benefits.  Or are you saying M-x
> grep is our top priority now?

I'm ambivalent on this issue, so take my word as an summary of my
understanding of the last discussion about it.

In other words, any improvement that tries to reduce the redundancy in
identifiers will require corresponding improvement in our tools to infer
the now-implicit information.

So we could bite the bullet and hope that someone will step up to the
plate and improve CEDET (or equivalent) to make up the difference.
Or we can sit here and wait for CEDET to improve first.
Or we can decide that this redundancy is really not that costly compared
to the benefits of being able to use dumb tools rather than being
dependent on more sophisticated packages.

When you said:

   That's a terrible idea. It makes it much harder to reason about the
   correct operation of programs, since now it makes it virtually
   impossible to implement find-definition functionality without lots of
   type inference.

you put your finger on the issue, indeed.  Using namespaces/modules
rather than Smalltalk-style objects would undoubtedly require
significantly less sophisticated analysis, but the same kind of tradeoff
is at play.

So for now, I insist on `gen-next' and `cl-flet'.  Even if it makes me
unpopular.

And note that I decided `cl-setf' was important/popular/frequent enough
to deserve to be promoted to `setf', and the same could happen at some
point to `gen-next' or `cl-flet'.


        Stefan

RE: Generators (iterators) for Gnu Emacs

[Drew Adams]

> The entire Common Lisp ecosystem is a counterexample so your rather
> sweeping declaration. In practice, namespaces are rather useful
> despite the need to *sometimes* fully *some* symbols. It's still
> better than having to fully specify *every* symbol on *every* use.

That was my experience also.

> > In practice, you have to write the package prefix whenever you
> > refer to a symbol that has one.  It might as well be part of the
> > symbol name itself.  Thus, packages complicate the language
> > definition while providing no benefit.
> 
> There's a lot of Lisp code out there that uses packages in the way
> they were meant to be used, and your personal distaste for the
> mechanism shouldn't keep namespaces out of elisp.

I wonder whether perhaps Richard ran into the need to use prefixes
so much because he was implementing the CL package system, and not
just making normal use of it.  Just a conjecture.

Re: Namespace macros

[Helmut Eller]

On Mon, Dec 08 2014, Artur Malabarba wrote:

> Names.el partially solves that by namespacing sharp-quoted symbols as
> function names (you have to use the sharp-quote on function names for this to
> work, but that's good practice anyway). This catches the majority of the uses
> of quotes for me.

For me define-key was a problematic and common case.  Until now I
believed that replacing 'COMMAND in (define-key ... 'COMMAND) with
#'COMMAND is inappropriate because I thought that #'F evaluates to
(symbol-function 'F).

But it turns out that my intuition was wrong and that (eq #'F 'F) => t
for all symbols F.  So #' can be used to mark function names in more
cases than I thought.  Which is good news for the namespace macro.

Helmut

Re: Namespace macros

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

>
> But it turns out that my intuition was wrong and that (eq #'F 'F) => t
> for all symbols F.  So #' can be used to mark function names in more
> cases than I thought.  Which is good news for the namespace macro.
>

Indeed, now a days #' is really just a note to the compiler. See for
instance
http://endlessparentheses.com/get-in-the-habit-of-using-sharp-quote.html

Below is a short example of a package using the macro, in case anyone's
interested. I particularly find it a pleasure to write and read code like
this.
https://github.com/Bruce-Connor/camcorder.el

[-- Attachment #2: Type: text/html, Size: 831 bytes --]

Re: Namespace macros

[Helmut Eller]

On Mon, Dec 08 2014, Artur Malabarba wrote:

> Below is a short example of a package using the macro, in case anyone's
> interested. I particularly find it a pleasure to write and read code like
> this.
> https://github.com/Bruce-Connor/camcorder.el

Yes, in most cases dropping the "camcorder-" prefix is a win.
But this part:

(define-minor-mode mode nil nil "sc"
  '(([f12] . camcorder-stop)
    ([f11] . camcorder-pause))
   ... )
   
doesn't look so great to me, because 1) "define-minor-mode mode" is
really useless for grep 2) it still uses the "camcorder-" prefix for key
bindings.

Helmut

Re: Generators (iterators) for Gnu Emacs

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

() Stefan Monnier <monnier@iro.umontreal.ca>
() Sun, 07 Dec 2014 22:23:58 -0500

   In other words, any improvement that tries to reduce the
   redundancy in identifiers will require corresponding
   improvement in our tools to infer the now-implicit
   information.

Right.  Sez Alan Perlis:

 Wherever there is modularity there is the potential for
 misunderstanding: Hiding information implies a need to
 check communication.

so the onerous activity here is the "check communication".
I would argue that the current FOO- scheme, being merely
convention, is actually more needful of inference than the
proposed FOO:: scheme.  If modularity is directly supported,
we (programs and humans, both) can turn from heuristics to
algorithms, w/ corresponding reduction in worry and angst
that the "check" is incomplete or incoherent.

That said, my experience (largely positive) is entirely w/
Guile Scheme modules, and i imagine that Emacs' modules would
be similar.  I don't know how Common Lisp does it -- maybe
someone who knows both systems could post a comparison?

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Namespace macros

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

>
> Yes, in most cases dropping the "camcorder-" prefix is a win.
> But this part:
>
> (define-minor-mode mode nil nil "sc"
>   '(([f12] . camcorder-stop)
>     ([f11] . camcorder-pause))
>    ... )
>
> doesn't look so great to me,

Yes. That package is a good example because it portrays how often it works
and how often it doesn't.
I personally find these small issues completely worth the overall beauty of
the code, but that's entirely a matter of preference.

FWIW, Both issues are workaroundable (I'm guessing that's not a word :-)).

> because 1) "define-minor-mode mode" is
> really useless for grep

This will be the case for any namespacing solution. You can work around by
defining the minor mode after the macro.
Also, I would be happy to add an option to avoid namespacing the
`define-minor-mode' form (and similar ones), if anyone wants it.

> 2) it still uses the "camcorder-" prefix for key
> bindings.

This can also be worked around by defining the keymap with a series of
`define-key' statements instead of doing it inside the `define-minor-mode'.
Which is something I see a lot of packages doing anyway.

Now that I think about it, I can also improve the macro to "look inside"
the fifth argument of a `define-minor-mode' call even if it's a quoted
list, since that argument is guaranteed to be a keymap.

[-- Attachment #2: Type: text/html, Size: 1679 bytes --]

Re: Namespace macros

[Dmitry Gutov]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> This will be the case for any namespacing solution. You can work around by
> defining the minor mode after the macro.

Not if the reader is namespace-aware. (See Clojure backquote).

Re: Namespace macros

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

On 8 Dec 2014 09:21, "Dmitry Gutov" <dgutov@yandex.ru> wrote:
>
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
> > This will be the case for any namespacing solution. You can work around
by
> > defining the minor mode after the macro.
>
> Not if the reader is namespace-aware. (See Clojure backquote).

I don't understand. What does that have to do with grep?

[-- Attachment #2: Type: text/html, Size: 574 bytes --]

Re: Namespace macros

[Dmitry Gutov]

On 12/08/2014 01:42 PM, Artur Malabarba wrote:

>  > > This will be the case for any namespacing solution. You can work
> around by
>  > > defining the minor mode after the macro.
>  >
>  > Not if the reader is namespace-aware. (See Clojure backquote).
>
> I don't understand. What does that have to do with grep?

Sorry, the above indeed doesn't make sense, and was speed-reading, and 
assumed the above referred to the item two.

What I meant to point out is that is the reader is namespace-aware, you 
can resolve the local names with special syntax, instead of making the 
namespace macro aware of `define-minor-mode' specifically.

Re: Namespace macros

[Helmut Eller]

On Mon, Dec 08 2014, Artur Malabarba wrote:

>> because 1) "define-minor-mode mode" is
>> really useless for grep
>
> This will be the case for any namespacing solution. You can work around by
> defining the minor mode after the macro.
> Also, I would be happy to add an option to avoid namespacing the
> `define-minor-mode' form (and similar ones), if anyone wants it.

Some commands like M-x gnus or M-x rmail don't have proper prefixes. I
suppose those would also need to be defined after the macro or with an
option to suppress automatic prefixing.

>> 2) it still uses the "camcorder-" prefix for key
>> bindings.
>
> This can also be worked around by defining the keymap with a series of
> `define-key' statements instead of doing it inside the `define-minor-mode'.
> Which is something I see a lot of packages doing anyway.

Something like `(([f12] . ,#'stop)) would probably also work.  Not 
exactly an improvement, though.

> Now that I think about it, I can also improve the macro to "look inside" the
> fifth argument of a `define-minor-mode' call even if it's a quoted list,
> since that argument is guaranteed to be a keymap.

I guess that temptation will come up often: just add a special case for
macro X.

Helmut

Re: Generators (iterators) for Gnu Emacs

[David Engster]

Stefan Monnier writes:
>>>> *Any* solution that frees developers from spelling all symbols in full
>>>> will have that disadvantage.
>>> That's right.
>> So what?  There are definitely offsetting benefits.  Or are you saying M-x
>> grep is our top priority now?
>
> I'm ambivalent on this issue, so take my word as an summary of my
> understanding of the last discussion about it.
>
> In other words, any improvement that tries to reduce the redundancy in
> identifiers will require corresponding improvement in our tools to infer
> the now-implicit information.
>
> So we could bite the bullet and hope that someone will step up to the
> plate and improve CEDET (or equivalent) to make up the difference.
> Or we can sit here and wait for CEDET to improve first.

CEDET has an Elisp parser. It's a bit difficult for me to extend it to
parse namespaces without knowing the syntax for that, but I can try.

Seriously: Yes, the parser could be improved. But it still would have
several drawbacks compared to grep et al.:

. It is slow.
. You need to set up some kind of project beforehand so that Semantic
  knows what to parse and where to find dependencies.
. It will only work inside Emacs.
. I guess that Changelog thingy will become more complicated.
. Did I mention it's slow?

I'm all for it.

-David

Re: Generators (iterators) for Gnu Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The entire Common Lisp ecosystem is a counterexample so your rather
  > sweeping declaration.

I have no way of knowing what is proved by Common Lisp programming.
I don't have time to study people's Common Lisp programs.

			  In practice, namespaces are rather useful despite
  > the need to *sometimes* fully *some* symbols. It's still better than
  > having to fully specify *every* symbol on *every* use.

In the Lisp Machine system, importing proved to be unusable.

If it is true that it is usable in some programs, the question is:
what are the circumstances that make them usable in some circumstances
and not in others?  In Emacs code, would they be usable or unusable?

I made a conjecture in another message just now.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Generators (iterators) for Gnu Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I wonder whether perhaps Richard ran into the need to use prefixes
  > so much because he was implementing the CL package system, and not
  > just making normal use of it.  Just a conjecture.

I implemented all the Common Lisp features (of which the package
system was one) in the Lisp Machine operating system, which included
lots of programs to do lots of things.  (An Emacs-like editor was one
of them.)

In practice, the goal of using names without prefixes via importing
namespace did not work out.

I make the conjecture that name space importing is convenient when you
have lots of small modules, each with its own name space and not many
names, and each referring to just a few other modules.  That's not how
the Lisp Machine system was written, and it's also mostly not how
Emacs was written.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Namespace macros

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Some commands like M-x gnus or M-x rmail don't have proper prefixes.

Could you explain what you mean?  The names 'gnus' and 'rmail' are fine.
They are commands to invoke packages with the same names.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

RE: Generators (iterators) for Gnu Emacs

[Drew Adams]

>> I wonder whether perhaps Richard ran into the need to use
>> prefixes so much because he was implementing the CL package
>> system, and not just making normal use of it.  Just a conjecture.
> 
> I implemented all the Common Lisp features (of which the package
> system was one) in the Lisp Machine operating system, which included
> lots of programs to do lots of things.  (An Emacs-like editor was
> one of them.)
> 
> In practice, the goal of using names without prefixes via importing
> namespace did not work out.
> 
> I make the conjecture that name space importing is convenient when
> you have lots of small modules, each with its own name space and not
> many names, and each referring to just a few other modules.  That's not
> how the Lisp Machine system was written, and it's also mostly not how
> Emacs was written.

Maybe.  Dunno what constitutes small, not many, few, etc.
Perhaps others can chime in with more relevant info here.
Sorry; I don't have anything concrete to offer in this regard.

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> Seriously: Yes, the parser could be improved. But it still would have
> several drawbacks compared to grep et al.:

> . It is slow.
> . You need to set up some kind of project beforehand so that Semantic
>   knows what to parse and where to find dependencies.
> . It will only work inside Emacs.
> . I guess that Changelog thingy will become more complicated.
> . Did I mention it's slow?

> I'm all for it.

As mentioned in my sample grep command, it'd have to parse all the emacs
and elpa Elisp files, in order to find all potential references.


        Stefan

Re: Generators (iterators) for Gnu Emacs

[Stefan Monnier]

> I make the conjecture that name space importing is convenient when you
> have lots of small modules, each with its own name space and not many
> names, and each referring to just a few other modules.  That's not how
> the Lisp Machine system was written, and it's also mostly not how
> Emacs was written.

In languages like SML, the common style is to always use qualified
names, tho the modules used in a file get locally renamed to use
shorter prefixes.


        Stefan

Re: Namespace macros

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On 8 Dec 2014 18:59, "Richard Stallman" <rms@gnu.org> wrote:
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Some commands like M-x gnus or M-x rmail don't have proper prefixes.
>
> Could you explain what you mean?  The names 'gnus' and 'rmail' are fine.
> They are commands to invoke packages with the same names.

He was just saying there's no way to define commands with these names
inside a Names.el namespace (because all definitions inside it are prefixed
with the package name).

He was just mentioning a limitation of Names.el, not criticising those
commands. (though, like I explained, it's not really a limitation).

[-- Attachment #2: Type: text/html, Size: 1024 bytes --]

Re: Namespace macros

[Helmut Eller]

On Mon, Dec 08 2014, Richard Stallman wrote:

>   > Some commands like M-x gnus or M-x rmail don't have proper prefixes.
>
> Could you explain what you mean?

Just that "gnus-" is not a prefix of "gnus".  If the prefix "gnus-" is
inserted automatically one would get names like "gnus-gnus" or perhaps
"gnus-".

> The names 'gnus' and 'rmail' are fine.

Sure.

Helmut

namespaces

[Ivan Shmakov]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

 >> I make the conjecture that name space importing is convenient when
 >> you have lots of small modules, each with its own name space and not
 >> many names, and each referring to just a few other modules.  That's
 >> not how the Lisp Machine system was written, and it's also mostly
 >> not how Emacs was written.

 > In languages like SML, the common style is to always use qualified
 > names, tho the modules used in a file get locally renamed to use
 > shorter prefixes.

	FWIW, my own preference for namespace-aware languages (that’s
	Scheme 48, Perl 5, and a few others) was always to import only
	what’s strictly necessary.

	Typically, of each module, only a few functions are used, so
	even without a prefix, they don’t really clutter the namespace.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A