streams are cool, you could stream virtually anything!

streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

Hi guys,

Aren't you tired of writing the same for loops in various contexts?
I know I was, and there's a better way!

I just added a function `stream-regexp' to stream.el to stream the
search of a regexp in a buffer:


    (let ((str (stream-regexp buf ".*foo\\(.*\\)$")))
      (seq-map (lambda (match-data)
                 ...)
               str))

(I still have to document it and test it properly)

Because streams are lazy and immutable, you can reuse such a stream
forever or keep it for later, move around in a buffer or modify the
buffer and re-evaluate the stream (you'll get new match data), or map
the result, etc.

All the credit goes to Damien Cassou for giving me the idea :-)

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> I just added a function `stream-regexp' to stream.el to stream the search of
> a regexp in a buffer:

This is amazing, Nicolas! I've wanted this exact construction so many times!

Do you know the memory behavior of doing it this way? That is, for 1000 hits,
how does the number of cons's allocated compare to the iterative approach?

John

Re: streams are cool, you could stream virtually anything!

[Damien Cassou]

John Wiegley <jwiegley@gmail.com> writes:
>> I just added a function `stream-regexp' to stream.el to stream the search of
>> a regexp in a buffer:
>
> This is amazing, Nicolas! I've wanted this exact construction so many times!
>
> Do you know the memory behavior of doing it this way? That is, for 1000 hits,
> how does the number of cons's allocated compare to the iterative approach?

I think it does not cost anything because nothing is stored in memory.
It's just a different way to write the while loop.

-- 
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: streams are cool, you could stream virtually anything!

[Phillip Lord]

At the risk of this sounding like a plug, it was my desire to get away
from looping that motivated my own library m-buffer
(https://github.com/phillord/m-buffer-el). This is not lazy, but is list
based.

I'd have a couple of questions with this approach. First, what happens
if the lambda *changes* the buffer? Mixing this form of mutability with
laziness is, I think, going to be unpredictable.

Second question, what happens to the match data? How would you deal with
nil'ing markers after use? Otherwise, you're going to fill the buffer up
with markers (they do GC, but slowly).

Phil

Nicolas Petton <nicolas@petton.fr> writes:

> Hi guys,
>
> Aren't you tired of writing the same for loops in various contexts?
> I know I was, and there's a better way!
>
> I just added a function `stream-regexp' to stream.el to stream the
> search of a regexp in a buffer:
>
>
>     (let ((str (stream-regexp buf ".*foo\\(.*\\)$")))
>       (seq-map (lambda (match-data)
>                  ...)
>                str))
>
> (I still have to document it and test it properly)
>
> Because streams are lazy and immutable, you can reuse such a stream
> forever or keep it for later, move around in a buffer or modify the
> buffer and re-evaluate the stream (you'll get new match data), or map
> the result, etc.
>
> All the credit goes to Damien Cassou for giving me the idea :-)
>
> Cheers,
> Nico

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Hi Nicolas,
> Because streams are lazy and immutable, you can reuse such a stream
> forever or keep it for later, move around in a buffer or modify the
> buffer and re-evaluate the stream (you'll get new match data), or map
> the result, etc.

A different cool thing you can use streams for is to traverse a file
tree recursively (to do something with the files, e.g. search them).

With streams, such an operation can be paused and resumed later.

Will make a PR if you are interested (but it is easy to implement).


Regards,

Michael.

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

John Wiegley <jwiegley@gmail.com> writes:

> Do you know the memory behavior of doing it this way? That is, for
> 1000 hits, how does the number of cons's allocated compare to the
> iterative approach?

That depends on whether you want/need to keep the matches so far.  If
you don't need to save prior matches, you can just

  (setq stream (stream-cdr stream))

after generating an element, and memory usage will be O(1).  (Note to
Nicolas: a `stream-pop' function would be nice in this context).

In that case, you can alternatively use generators (generator.el) or the
derived iterators.el.

If you keep the complete stream in memory, you get O(nbr-matches) conses
(of course).  Not much a problem IME, if you keep an eye on what you
want to keep and what you can throw away when using streams.


Regards,

Michael.

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> after generating an element, and memory usage will be O(1).

If you iterate you will produce O(nbr-matches) new conses of course.
But as had been said, that's not different from a (regular) loop.


Michael.

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> That depends on whether you want/need to keep the matches so far.  If
> you don't need to save prior matches, you can just
>
>   (setq stream (stream-cdr stream))
>
> after generating an element, and memory usage will be O(1).  (Note to
> Nicolas: a `stream-pop' function would be nice in this context).

Indeed!  Could I by any chance get another pull request from you on
GitHub?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 355 bytes --]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> I'd have a couple of questions with this approach. First, what happens
> if the lambda *changes* the buffer? Mixing this form of mutability with
> laziness is, I think, going to be unpredictable.

I agree, but I don't see how that's any different from modifying the
buffer while in a while loop?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> A different cool thing you can use streams for is to traverse a file
> tree recursively (to do something with the files, e.g. search them).
>
> With streams, such an operation can be paused and resumed later.
>
> Will make a PR if you are interested (but it is easy to implement).

Of course, that an awesome idea, thank you very much!

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

As more of these patterns emerge, I'm even more convinced that  the
stream package belongs alongside thunk.el in core emacs.
-- 

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

"T.V Raman" <raman@google.com> writes:

> As more of these patterns emerge, I'm even more convinced that  the
> stream package belongs alongside thunk.el in core emacs.

I think so too, but I'm obviously biased :)

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Nicolas Petton <nicolas@petton.fr> writes:

> Indeed!  Could I by any chance get another pull request from you on
> GitHub?

Of course.  Are you ok when `stream-pop' would raise a (newly defined)
error when called on an empty stream?

The problem is how one can notice when a stream has run out of elements
when calling `stream-pop'.  We could either just return nil (bad,
because the stream could also generate nil if non-empty), return a
dedicated flag (like an uninterned symbol that is internal to
stream.el), or just signal a dedicated error that the programmer has to
catch.  I vote for the third alternative.


Michael.

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> T V Raman <raman@google.com> writes:

> As more of these patterns emerge, I'm even more convinced that the stream
> package belongs alongside thunk.el in core emacs.

Let's see it develop during these earlier stages on ELPA, so people can easily
update the package as it develops. Once it stabilizes, I agree we should move
it into core.

John

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

SGTM.

While on the topic -- should we move things out of core into elpa --
there are a lot of elisp packages that are presently bundled -- mostly
because "there was no elpa around when they were bundled".

John Wiegley writes:
 > >>>>> T V Raman <raman@google.com> writes:
 > 
 > > As more of these patterns emerge, I'm even more convinced that the stream
 > > package belongs alongside thunk.el in core emacs.
 > 
 > Let's see it develop during these earlier stages on ELPA, so people can easily
 > update the package as it develops. Once it stabilizes, I agree we should move
 > it into core.
 > 
 > John

-- 

-- 

ELPA policy (Was: streams are cool, you could stream virtually anything!)

[John Wiegley]

>>>>> T V Raman <raman@google.com> writes:

> While on the topic -- should we move things out of core into elpa -- there
> are a lot of elisp packages that are presently bundled -- mostly because
> "there was no elpa around when they were bundled".

I would in fact like to see more packages move from core into ELPA, although
I'm not eager to do so willy nilly. Some packages now in Core are just fun, or
should be there because people are used to them being there, even if today we
wouldn't necessarily add that package to core.

If someone would be willing to go through every core package, and produce a
list of what should stay in core, what should move to ELPA, and why they think
so, that would be a great starting point.

I know from past experience that reading every core file is highly interesting
and educational, and I guarantee whoever volunteers for this will discover
many cool and interesting things. :)

John

ELPA policy (Was: streams are cool, you could stream virtually anything!)

[T.V Raman]

Reading through those files is how I taught myself   enough  about
Emacs to write emacspeak:-)

John Wiegley writes:
 > >>>>> T V Raman <raman@google.com> writes:
 > 
 > > While on the topic -- should we move things out of core into elpa -- there
 > > are a lot of elisp packages that are presently bundled -- mostly because
 > > "there was no elpa around when they were bundled".
 > 
 > I would in fact like to see more packages move from core into ELPA, although
 > I'm not eager to do so willy nilly. Some packages now in Core are just fun, or
 > should be there because people are used to them being there, even if today we
 > wouldn't necessarily add that package to core.
 > 
 > If someone would be willing to go through every core package, and produce a
 > list of what should stay in core, what should move to ELPA, and why they think
 > so, that would be a great starting point.
 > 
 > I know from past experience that reading every core file is highly interesting
 > and educational, and I guarantee whoever volunteers for this will discover
 > many cool and interesting things. :)
 > 
 > John

-- 

-- 

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Nicolas Petton <nicolas@petton.fr> writes:
>
>> Indeed!  Could I by any chance get another pull request from you on
>> GitHub?
>
> Of course.  Are you ok when `stream-pop' would raise a (newly defined)
> error when called on an empty stream?

Yup, that's fine with me.

One question though, how do you plan to implement it? I ask because now
that I think about it, streams have been (until now) fully immutable.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Artur Malabarba]

2015-11-05 9:27 GMT+00:00 Nicolas Petton <nicolas@petton.fr>:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> Nicolas Petton <nicolas@petton.fr> writes:
>>
>>> Indeed!  Could I by any chance get another pull request from you on
>>> GitHub?
>>
>> Of course.  Are you ok when `stream-pop' would raise a (newly defined)
>> error when called on an empty stream?
>
> Yup, that's fine with me.
>
> One question though, how do you plan to implement it? I ask because now
> that I think about it, streams have been (until now) fully immutable.

I would expect it to be a macro,
(prog1 (stream-car stream)  (setq stream (stream-cdr stream)))

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I would expect it to be a macro,
> (prog1 (stream-car stream)  (setq stream (stream-cdr stream)))

My thought exactly.  Do you think this should be in stream.el?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Phillip Lord]

Nicolas Petton <nicolas@petton.fr> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> I'd have a couple of questions with this approach. First, what happens
>> if the lambda *changes* the buffer? Mixing this form of mutability with
>> laziness is, I think, going to be unpredictable.
>
> I agree, but I don't see how that's any different from modifying the
> buffer while in a while loop?


Emacs is single threaded, so the only thing that can change the buffer
during a while loop is the stuff in the while loop. So, with m-buffer,
for instance, you get back a list of markers based on the world now.

With laziness, though, do the results of the stream depend on the state
of the buffer when the stream is created, realised, or the state of the
buffer now.

I don't know if this is a problem or not, which is why I am asking.

Phil

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Emacs is single threaded, so the only thing that can change the buffer
> during a while loop is the stuff in the while loop. So, with m-buffer,
> for instance, you get back a list of markers based on the world now.
>
> With laziness, though, do the results of the stream depend on the state
> of the buffer when the stream is created, realised, or the state of the
> buffer now.

It depends on the state of the buffer during the execution of the
mapping in my example, i.e. when the stream is realized, so the behavior
is similar to the while loop.

> I don't know if this is a problem or not, which is why I am asking.

No, it's not, unless you do things like (`seq-do' is eager):

    (seq-do (lambda (match-data)
               (something-that-modifies-the-buffer)
               ...)
             (stream-regexp buf ".*foo\\(.*\\)$")))))

But then it's just the same issue as with a while loop:

     (cl-loop
       while (re-search-forward "^  - \\(.*\\) :: \\(.*\\)$" end t)
       do (something-that-modifies-the-buffer)
       ...)

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I would expect it to be a macro,
> (prog1 (stream-car stream)  (setq stream (stream-cdr stream)))

Exactly, that's what I intent.


Michael.

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Nicolas Petton <nicolas@petton.fr> writes:

> > I don't know if this is a problem or not, which is why I am asking.
>
> No, it's not, unless you do things like (`seq-do' is eager):

Let's say it depends on what you want to achieve with this function.

The current implementation is simple but has two downsides:

  - It pollutes the searched buffer with markers, unnecessarily, because
  it doesn't allow to change the buffer, so positions (numbers) would
  suffice

  - It has no state.  If you want to use it like an iterator (aka
  `stream-pop'), you currently can't pause, do something else (move
  point, edit the buffer), and continue.

Here is a version that tries to solve these problems:

--8<---------------cut here---------------start------------->8---
;; -*- lexical-binding: t -*-
(defun stream-regexp (buffer regexp)
  (let ((match nil))
    (letrec ((builder
              (lambda (buffer regexp)
                (stream-make
                 (save-match-data
                   (save-excursion
                     (goto-char (or match (point-min)))
                     (with-current-buffer buffer
                       (setq match (re-search-forward regexp nil t)))
                     (when match
                       (setq match (copy-marker match))
                       (cons (cons (match-beginning 0)
                                   (match-end 0))
                             (funcall builder buffer regexp)))))))))
      (funcall builder buffer regexp))))
--8<---------------cut here---------------end--------------->8---



Regards,

Michael.

Re: streams are cool, you could stream virtually anything!

[raman]

1+ on putting this in stream.el -- call it stream-cdr and alias
stream-first to stream-car; other name possibilities stream-head and
stream-tail.

I dont like stream-pop because users might then expect there to be a
stream-push --- which might cause confusion.



-- 

Re: streams are cool, you could stream virtually anything!

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 05 Nov 2015 12:28:42 +0000
> Cc: Damien Cassou <damien@cassou.me>, emacs-devel <emacs-devel@gnu.org>
> 
> Emacs is single threaded, so the only thing that can change the buffer
> during a while loop is the stuff in the while loop.

Apologies if what I'm saying makes no sense for this particular
discussion (I didn't track it), but such assumptions are generally
incorrect, unless you have extremely tight control on the code inside
that loop (read: you yourself wrote every single function it calls,
and you know very well what you are doing).  Otherwise, you might be
in for a surprise.  For example, code that calls sit-for or even
read-event can invoke timers, which can do all kinds of things to your
buffer.  Even some primitive that calls QUIT could potentially do
that, although not with the current code base (QUIT invokes atimers).

There be dragons.

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

raman <raman@google.com> writes:

> I dont like stream-pop because users might then expect there to be a
> stream-push --- which might cause confusion.

Actually I plan a `stream-push' - why not?  There is nothing wrong with
that; `push' itself doesn't modify any list, it just sets the PLACE to
the newly created cons
  
  (cons NEWELT PLACE)

We can do the very same for streams; just use `stream-cons' instead of
`cons':

--8<---------------cut here---------------start------------->8---
(defmacro stream-push (newelt place)
  "Add NEWELT to the stream stored in PLACE.
This is equivalent to (setf PLACE (stream-cons NEWELT PLACE))."
  `(cl-callf2 stream-cons ,newelt ,place))
--8<---------------cut here---------------end--------------->8---

Would that be ok?


Regards,

Michael

streams and generators (was: streams are cool, you could stream virtually anything!)

[Michael Heerdegen]

John Wiegley <jwiegley@gmail.com> writes:

> Let's see it develop during these earlier stages on ELPA, so people
> can easily update the package as it develops. Once it stabilizes, I
> agree we should move it into core.

I want to draw our attention to "generator.el" also, because when we
think about the future direction, we should keep in mind both of them,
because they are very related.

generator.el uses an implicit continuation passing style to define
`defgenerator' - as far as I understand it, the code is really clever,
an excellent piece of Elisp.  It deserves more attention (and some
broader documentation).

We should think about how the two can benefit from each other.  For
example there should be a way to convert a generator to a stream etc.

I already wrote "iterators.el" (Elpa), a trivial thing compared to
"generator.el", that implements some stream like basic operations for
generators/iterators.  All that stuff should fit together at the end.


Regards,

Michael.

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 562 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> We can do the very same for streams; just use `stream-cons' instead of
> `cons':
>
> --8<---------------cut here---------------start------------->8---
> (defmacro stream-push (newelt place)
>   "Add NEWELT to the stream stored in PLACE.
> This is equivalent to (setf PLACE (stream-cons NEWELT PLACE))."
>   `(cl-callf2 stream-cons ,newelt ,place))
> --8<---------------cut here---------------end--------------->8---
>
> Would that be ok?

I guess so, as long as we don't add too many such functions.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams and generators

[Dmitry Gutov]

On 11/05/2015 07:06 PM, Michael Heerdegen wrote:
> I want to draw our attention to "generator.el" also, because when we
> think about the future direction, we should keep in mind both of them,
> because they are very related.

Agreed. I haven't really tried it yet, but it seems to be one of the 
most interesting of the sequence-related additions.

I can see us adding support in xref to accept generators as well, where 
we now only pass a list of xrefs. This should eliminate the main 
advantage rgrep has over xref-find-regexp: displaying matches as soon as 
Grep outputs each one.

Re: streams are cool, you could stream virtually anything!

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On 5 Nov 2015 9:58 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
>
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
> > We can do the very same for streams; just use `stream-cons' instead of
> > `cons':
> >
> > --8<---------------cut here---------------start------------->8---
> > (defmacro stream-push (newelt place)
> >   "Add NEWELT to the stream stored in PLACE.
> > This is equivalent to (setf PLACE (stream-cons NEWELT PLACE))."
> >   `(cl-callf2 stream-cons ,newelt ,place))
> > --8<---------------cut here---------------end--------------->8---
> >
> > Would that be ok?
>
> I guess so, as long as we don't add too many such functions.

I dunno. The way I think of streams it makes perfect sense to have a pop
without a push. But maybe that's just me. You could call it stream-next or
something like that, so people won't be confused by the lack of push.

[-- Attachment #2: Type: text/html, Size: 1228 bytes --]

Re: streams are cool, you could stream virtually anything!

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

On 5 Nov 2015 11:52 am, "Nicolas Petton" <nicolas@petton.fr> wrote:
>
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
> > I would expect it to be a macro,
> > (prog1 (stream-car stream)  (setq stream (stream-cdr stream)))
>
> My thought exactly.  Do you think this should be in stream.el?

Yes. It's a super convenient way to consume sequences.

[-- Attachment #2: Type: text/html, Size: 571 bytes --]

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

I still  like these things to be in pairs, and I'm not particularly
sold on having stream-pop.

I like the symmettry in stream-head and stream-tail --  probably
because I originally learnt streams from SICP.


Artur Malabarba writes:
 > On 5 Nov 2015 9:58 pm, "Nicolas Petton" <nicolas@petton.fr> wrote:
 > >
 > > Michael Heerdegen <michael_heerdegen@web.de> writes:
 > >
 > > > We can do the very same for streams; just use `stream-cons' instead of
 > > > `cons':
 > > >
 > > > --8<---------------cut here---------------start------------->8---
 > > > (defmacro stream-push (newelt place)
 > > > "Add NEWELT to the stream stored in PLACE.
 > > > This is equivalent to (setf PLACE (stream-cons NEWELT PLACE))."
 > > > `(cl-callf2 stream-cons ,newelt ,place))
 > > > --8<---------------cut here---------------end--------------->8---
 > > >
 > > > Would that be ok?
 > >
 > > I guess so, as long as we don't add too many such functions.
 > 
 > I dunno. The way I think of streams it makes perfect sense to have a pop without a push. But maybe that's just me. You could call it stream-next or something like that, so people won't be confused by the lack of push. 
 > 

-- 

-- 

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I dunno. The way I think of streams it makes perfect sense to have a
> pop without a push. But maybe that's just me. You could call it
> stream-next or something like that, so people won't be confused by the
> lack of push.

I'm ok with not adding a push, I'm too not sure if it could ever be
useful.

OTOH, it has advantages to keep the name `pop': it's easy to discover,
and if there are people requesting it for good reasons, we can still add
it later without having to rename the other thing to "pop" again.

Streams are analogous to lists in many regards, but are different in
some aspects too, so I don't think it's a problem per se when there are
functions named analogously, and related functions are different or
named different or missing.

If you name it `stream-next', I guess most people will wonder anyway
"ok, that's probably just pop, but where's the push, or does that make
no sense?".  And when they use it the second time, they will wonder
"Mmh, I know it was _not_ named `stream-pop' - but what was the name
again?".


Michael.

Re: streams are cool, you could stream virtually anything!

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On 5 Nov 2015 11:34 pm, "T.V Raman" <raman@google.com> wrote:
>
> I still  like these things to be in pairs, and I'm not particularly
> sold on having stream-pop.

Perhaps slightly unrelated, but I don't even think of push pop as a pair. I
mean, I see the obvious the relation. It's just that I've never even used
them in the same function, even though I use both a lot.

But I digress. I haven't even used stream.el yet, so I'm just assuming
stream-pop would be as useful as pop.

[-- Attachment #2: Type: text/html, Size: 638 bytes --]

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

The mental pairing of push, pop isn't so much about using them in the
same function -- it's more about your mental model of the API and what
you expect  -- so if you see stream-pop as an exported API function
--- you implicitly assume that there must be a stream-push 

Artur Malabarba writes:
 > On 5 Nov 2015 11:34 pm, "T.V Raman" <raman@google.com> wrote:
 > >
 > > I still like these things to be in pairs, and I'm not particularly
 > > sold on having stream-pop.
 > 
 > Perhaps slightly unrelated, but I don't even think of push pop as a pair. I mean, I see the obvious the relation. It's just that I've never even used them in the same function, even though I use both a lot. 
 > 
 > But I digress. I haven't even used stream.el yet, so I'm just assuming stream-pop would be as useful as pop. 
 > 

-- 

-- 

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

Hi Nicolas,

A rich source of ideas could come from looking at the streaming libraries of
other languages, like SERIES in Common Lisp, or conduit in Haskell. To give
just a few ideas of what might arise from such research:

    stream-take N STR

        Produce a stream composed of up to the first N elements of STR.

    stream-drop N STR

        Produce a stream starting after the first N elements of STR.

        stream-cdr = stream-drop 1

    stream-apply #'FUNC STR1.. STRN

        Given a function, reads its arguments from STR1.. STRN, and #'apply
        the function to those arguments. Its result is the corresponding
        element of the resulting stream. Continue until one of the argument
        streams reaches its end.

        stream-zip = stream-apply #'cons
    
    stream-apply* #'FUNC STR1.. STRN

        Like stream-apply, but any exhausted argument stream is seen to return
        nil until all are exhausted.

    stream-iterate FUNC VAL

        Produce a stream by funcall'ing FUNC on VAL to determine the first
        element, then funcall FUNC on that result to produce the next element,
        so on until infinity.

    stream-isolate N STR

        Ensure a resulting stream that is exactly N elements long. This is
        like stream-take if STR is longer, otherwise it's like appending N-M
        nil values onto STR, where M is the actual length.

    stream-append STR1 STR2

        Concatenate two streams, producing a longer one.

    stream-loop

        This is a macro wrapper around `loop' that adds a new keyword:
        streaming. Like collect, except it collects the results in a stream
        instead of a list.

    stream-list LIST

        Make a list look like a stream (i.e., accessible using the stream
        API).

    stream-collect STR

        Drain all the elements of STR into a list.

    stream-repeat N VAL

        A stream consisting of VAL repeated N times.

This list could easily be many times as long... For more inspiration see:

    http://hackage.haskell.org/package/conduit-combinators-1.0.3/docs/Conduit.html

If you'd like me continue, just say. I've written libraries similar to this
before, just not in Emacs Lisp (and it's awesome that it's becoming easy to!).

John
        

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> Michael Heerdegen <michael_heerdegen@web.de> writes:

> OTOH, it has advantages to keep the name `pop': it's easy to discover, and
> if there are people requesting it for good reasons, we can still add it
> later without having to rename the other thing to "pop" again.

I'd prefer `stream-car', myself. Maintaining consistency of nomenclature
between "in memory lists" and "generated lists" (aka, streams) makes it very
intuitive to pick the right name. And there will be many new ones to come.
This is a mine rich and waiting.

John

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

John Wiegley <jwiegley@gmail.com> writes:

> I'd prefer `stream-car', myself. Maintaining consistency of
> nomenclature between "in memory lists" and "generated lists" (aka,
> streams) makes it very intuitive to pick the right name. And there
> will be many new ones to come.  This is a mine rich and waiting.

Are you really speaking about what had been discussed here,
i.e. somewhat like this:

--8<---------------cut here---------------start------------->8---
(defmacro stream-pop (place)
  "Return the first element of the stream stored in PLACE.
Set PLACE to it's `stream-cdr'."
  `(if (stream-empty-p ,place)
       (signal 'some-new-error-type nil)
     (prog1 (stream-first ,place)
       (cl-callf stream-rest ,place))))
--8<---------------cut here---------------end--------------->8---

or about what is currently named `stream-first'?  When you want the
above pop thing to be named `stream-car', I don't understand your
preference.

If not: FWIW, personally I prefer `stream-first' and `stream-rest' to
`stream-car' and `stream-cdr' because these different names underline
that streams are not conses.  And that's an important aspect to keep in
mind.  In particular, streams are immutable, but a function named
`stream-car' would somehow suggest the opposite.  Not only are names
important to be mnemonic; they are also a way for the user to build a
mental model, and that should not lead him to problematic analogies.


Michael.

Re: streams are cool, you could stream virtually anything!

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Thu, 05 Nov 2015 12:28:42 +0000
>> Cc: Damien Cassou <damien@cassou.me>, emacs-devel <emacs-devel@gnu.org>
>> 
>> Emacs is single threaded, so the only thing that can change the buffer
>> during a while loop is the stuff in the while loop.
>
> Apologies if what I'm saying makes no sense for this particular
> discussion (I didn't track it), but such assumptions are generally
> incorrect, unless you have extremely tight control on the code inside
> that loop (read: you yourself wrote every single function it calls,
> and you know very well what you are doing).  Otherwise, you might be
> in for a surprise.  For example, code that calls sit-for or even
> read-event can invoke timers, which can do all kinds of things to your
> buffer.  Even some primitive that calls QUIT could potentially do
> that, although not with the current code base (QUIT invokes atimers).
>
> There be dragons.


Eli

You are, indeed, correct. Perhaps I could rephrase, and say that during
a while loop there are circumstances when the buffer could change as a
result of something else happening.

With a lazily evaluation of a search through the buffer, though, I think
that those issues might be larger.

Still, you can never tell with these things for sure, till you try them.
A streaming interface to emacs buffers sound like it might be
interesting, and I've thought about it myself before. We shall see!

Phil

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:
>
> I'm ok with not adding a push, I'm too not sure if it could ever be
> useful.

I agree.

> OTOH, it has advantages to keep the name `pop': it's easy to discover,
> and if there are people requesting it for good reasons, we can still add
> it later without having to rename the other thing to "pop" again.

Yes, `stream-pop' is ok with me too.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

John Wiegley <jwiegley@gmail.com> writes:


> I'd prefer `stream-car', myself.

Would an alias to the trick?  On one hand an alias is cheap, but on the
other hand I don't want to add too many aliases as I don't want to
pollute stream's API.  In this case aliasing would make sense though I
think.

Nico


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On 6 Nov 2015 10:49 am, "Nicolas Petton" <nicolas@petton.fr> wrote:
>
> John Wiegley <jwiegley@gmail.com> writes:
>
> > I'd prefer `stream-car', myself.
>
> Would an alias to the trick?

Are we still taking about the pop operation here? Because I would expect
stream-car to not modify the variable.

[-- Attachment #2: Type: text/html, Size: 498 bytes --]

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 6 Nov 2015 10:49 am, "Nicolas Petton" <nicolas@petton.fr> wrote:
>>
>> John Wiegley <jwiegley@gmail.com> writes:
>>
>> > I'd prefer `stream-car', myself.
>>
>> Would an alias to the trick?
>
> Are we still taking about the pop operation here? Because I would expect
> stream-car to not modify the variable.

No, I guess that was for `stream-first' and `stream-rest', right?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Filipp Gunbin]

On 06/11/2015 03:47 +0100, Michael Heerdegen wrote:

[...]

> If not: FWIW, personally I prefer `stream-first' and `stream-rest' to
> `stream-car' and `stream-cdr' because these different names underline
> that streams are not conses.  And that's an important aspect to keep in
> mind.  In particular, streams are immutable, but a function named
> `stream-car' would somehow suggest the opposite.

`...-car' and `...-cdr' suggests only the presence of car and cdr (or
head and tail), not the mutability of these, and there could also be
`set-first' and `set-rest', although the latter sounds awkward.

SICP talks about lists and streams, showing they are different
approaches to sequences, and defines similar operations on both of them
- I think having car and cdr for both is a nice analogy.

Filipp

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

> Hi Nicolas,

Hi John,

> A rich source of ideas could come from looking at the streaming libraries of
> other languages, like SERIES in Common Lisp, or conduit in Haskell. To give
> just a few ideas of what might arise from such research:
>
>     stream-take N STR
>
>         Produce a stream composed of up to the first N elements of STR.
>
>     stream-drop N STR
>
>         Produce a stream starting after the first N elements of STR.
>
>         stream-cdr = stream-drop 1
>
>     stream-apply #'FUNC STR1.. STRN
>
>         Given a function, reads its arguments from STR1.. STRN, and #'apply
>         the function to those arguments. Its result is the corresponding
>         element of the resulting stream. Continue until one of the argument
>         streams reaches its end.
>
>         stream-zip = stream-apply #'cons

Just to be sure, have you seen that streams are "seq"-eables? It means
that `seq-take', `seq-drop', `seq-map', `seq-filter', `seq-reduce',
etc. all work on streams (with lazy evaluation as well of course).

Some of the functions you mentioned are nice though.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> No, I guess that was for `stream-first' and `stream-rest', right?

Correct, I didn't mean to talk about stream-pop/push, but first/rest.

Since first/rest do mean car/cdr in cl.el, having both sets makes sense. But I
also meant to comment on how we choose more streaming names in the future.

As someone else mentioned, lists and streams overlap at the notion of
sequences -- the main differences being the operational semantics of laziness,
and the semantics of allowing infinite streams.

So let's say I've written a function over lists, and I realize I want it to
act on streams instead. I'd hope that I could just insert "stream-" in several
places and fulfill that need. If I have to also change over to a different set
of names for the same effective operations, that would be unfortunate.

One could even imagine a wrapper library that works on both lists and streams
-- since lists are effectively pre-computed streams -- but that may be more
than we need just yet.

John

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> Just to be sure, have you seen that streams are "seq"-eables? It means that
> `seq-take', `seq-drop', `seq-map', `seq-filter', `seq-reduce', etc. all work
> on streams (with lazy evaluation as well of course).

No! I'm not very familiar with `seq' yet, clearly. Thanks for letting me know.
I need to bone up on what Emacs is capable of these days. Time to read the
manual and the lisp/ directory again. :)

John

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> John Wiegley <jwiegley@gmail.com> writes:

> As someone else mentioned, lists and streams overlap at the notion of
> sequences -- the main differences being the operational semantics of
> laziness, and the semantics of allowing infinite streams.

> So let's say I've written a function over lists, and I realize I want it to
> act on streams instead. I'd hope that I could just insert "stream-" in
> several places and fulfill that need. If I have to also change over to a
> different set of names for the same effective operations, that would be
> unfortunate.

This got sent before reading your comments on `seq', Nicolas, so please
disregard.

John

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

> No! I'm not very familiar with `seq' yet, clearly. Thanks for letting me know.
> I need to bone up on what Emacs is capable of these days. Time to read the
> manual and the lisp/ directory again. :)

Indeed, you'll even find a chapter in the elisp manual about this in the
sequences section!

    (info "(elisp) Sequences Arrays Vectors")

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

John Wiegley <jwiegley@gmail.com> writes:


> So let's say I've written a function over lists, and I realize I want it to
> act on streams instead. I'd hope that I could just insert "stream-" in several
> places and fulfill that need. If I have to also change over to a different set
> of names for the same effective operations, that would be unfortunate.

That's exactly why I did seq.el! :-)

It doesn't have `seq-first' and `seq-rest' though, but I could very well
add them.  The only thing we will have to know is that `seq-rest' will
be more expensive on arrays than lists and streams.

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
> I would in fact like to see more packages move from core into ELPA, although
> I'm not eager to do so willy nilly. Some packages now in Core are just fun, or
> should be there because people are used to them being there, even if today we
> wouldn't necessarily add that package to core.

I've proposed this before, but I guess I should run it by you again (and
warn you it wasn't favorably received):

The current definition of "core" is that the sources live inside the
Emacs repository.  Several of the larger core packages like Org, CEDET
and Gnus are already largely developed outside Emacs, for instance
because the developers want to keep them compatible with different/older
Emacsen and need to be synced into Emacs sources to make them "core"
anyway.

I posit that the only thing that actually matters for something to be
considered "core" is that authors of other packages can rely on the
(stable) API provided by these packages to be available in an Emacs as
it gets distributed and no installation of further packages or software
is necessary, neither by the sysadmin nor the user.  If so, instead of
keeping the "core" sources all in Emacs, they could equally well be
living in ELPA and be pre-installed into the distribution, or installed
into the Emacs build tree as submodules or subtrees.  The most radical
(and likely most controversial) thing to do would be to move everything
to ELPA that isn't needed to bootstrap Emacs.

Doing this would need some as of yet non-existing infrastructure to get
the chosen ELPA version of each package built into the distribution, and
facilities for sysadmins and users to update (but not disable) the
"core" packages at the system level or in their private directories.
Emacs may eventually distribute some "non-core" packages also that
sysadmins or users could disable if they chose to not use them.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: ELPA policy

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Sun, 08 Nov 2015 18:18:26 +0100
> 
> I posit that the only thing that actually matters for something to be
> considered "core" is that authors of other packages can rely on the
> (stable) API provided by these packages to be available in an Emacs as
> it gets distributed and no installation of further packages or software
> is necessary, neither by the sysadmin nor the user.  If so, instead of
> keeping the "core" sources all in Emacs, they could equally well be
> living in ELPA and be pre-installed into the distribution, or installed
> into the Emacs build tree as submodules or subtrees.

IMO, no serious move such as this one should be argued for, let alone
attempted, without some minimal analysis of advantages and
disadvantages.  In particular, such an analysis cannot be limited to
the POV of maintainers of packages tat are currently not bundled, it
must first and foremost look at this from the POV of the Emacs
maintenance, definitely if the argument is to leave in the Emacs
repository only what's needed for bootstrap.

And any change in maintenance routine, small or large, should have
enough advantages to justify the energy that will certainly go into
the move itself and into cleaning up the resulting fallout.

I don't see how we can seriously discuss such suggestions when they
are not accompanied by anything like the analysis they need.

> The most radical (and likely most controversial) thing to do would
> be to move everything to ELPA that isn't needed to bootstrap Emacs.

Most such packages don't have any active maintainers, i.e. they are
maintained by the "FSF", which means us the core developers.  IMO, it
makes very little sense to spread the stuff we maintain between 2
separate repositories, because all this does is add overhead and
complexity without any clear benefits that I could see.

Another important aspect that this suggestion seems to overlook is
that Emacs packages rely on others not only via APIs, but also by
inheritance, like all the modes that derive from Text mode etc.

> Doing this would need some as of yet non-existing infrastructure to get
> the chosen ELPA version of each package built into the distribution, and
> facilities for sysadmins and users to update (but not disable) the
> "core" packages at the system level or in their private directories.

Yes, and the effort this will require is squarely in the disadvantages
camp.

Let me turn the table and ask: Are there any _advantages_ in moving
stuff like Dired, CC Mode, Shell Mode, Speedbar, and ps-print, to name
just a random few?

Re: ELPA policy

[Aaron Ecay]

Hi Eli,

2015ko azaroak 8an, Eli Zaretskii-ek idatzi zuen:

[...]

> 
> Let me turn the table and ask: Are there any _advantages_ in moving
> stuff like Dired, CC Mode, Shell Mode, Speedbar, and ps-print, to name
> just a random few?

Imagine someone implements an awesome new feature for dired.  Emacs
users the world over are amazed by this, and fill their blogs, twitter,
etc. with the news.  If dired is an ELPA package, everyone who hears
this news can get the new feature in their emacs instantly by upgrading
their ELPA packages.  No need to wait N months for a new release of
emacs, or compile a non-release version of emacs from git.

Whether this counts as an advantage or not is complicated, and partially
depends on one’s point of view.  It would spell the end of an era where
each emacs release’s features are explicated through a year of excited
blogposts (for example
<http://endlessparentheses.com/tags-expanded.html#emacs-25>).  Users
will be able to choose a new model where features trickle in rather than
coming in sudden major chunks.  This might make it more complicated to
advocate adoption of new emacs versions.  But I don’t think so actually:
we regularly get big features at the language level too, which are very
exciting (for emacs 24 lexical binding, now dynamic modules and
xwidgets, and one dares to hope for the concurrency branch in emacs26).

I also think that discussion around this topic is distorted by a
long-tailed distribution.  Most people probably have in mind the big
emacs packages with dynamic developer communities and independent
release cycles.  Org, Gnus, CEDET, and maybe a few others.  On the other
hand, it seems to me that you have in mind (in addition to these) all
the tiny packages that see very few commits and perhaps no new features
(in the extreme case take kermit.el, which has seen no substantive
changes since at least 1992, but still gets its copyright header
lovingly updated every year).  I don’t know how to reconcile these
viewpoints, or even if it’s necessary.  Just something to consider.

Speaking for myself, my primary interest in emacs development is working
on org-mode, and I heavily use org as well as third-party packages for
python (elpy), clojure (cider), R (ESS), and a few other random things.
I build emacs from git every month or so in order to pick up little
quality of life improvements, like with-eval-afer-load, some of the
subr-x stuff, seq.el, the overhauled package menu, etc.  But I have to
make sure to keep a couple months’ worth of old emacsen around, in case
my monthly pull happens to land on a buggy commit – one that causes
regular segfaults, which I have had happen more than once.  I use emacs
for work so I don’t have the luxury of just going without for a few days
until the problem is fixed.  I could go back to a previous released
version of course, but then my init.el breaks because it relies on new
features since the release.

I’d be much happier running released versions of emacs for stability, if
it were easier to pull in new versions of core elisp libraries as they
are developed.  I suspect there are some others out there like me, and
many more who don’t currently build emacs from source for whatever reason
but would enjoy more regular access to new features in core elisp
libraries.

My 2 cents FWIW,
Aaron

PS I also have the impression that life would be easier for org’s
maintainers under a system that freed them from merging code back and
forth from org’s repo to emacs.  This would have indirect positive
effects on me as I work on org development.  However, I’m sure the
maintainers themselves – and maintainers of other projects in the same
situation like CEDET and Gnus – can speak to the question better than I
can.

-- 
Aaron Ecay

Re: ELPA policy

[Eli Zaretskii]

> From: Aaron Ecay <aaronecay@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 08 Nov 2015 20:52:12 +0000
> 
> Imagine someone implements an awesome new feature for dired.  Emacs
> users the world over are amazed by this, and fill their blogs, twitter,
> etc. with the news.  If dired is an ELPA package, everyone who hears
> this news can get the new feature in their emacs instantly by upgrading
> their ELPA packages.  No need to wait N months for a new release of
> emacs, or compile a non-release version of emacs from git.

How is this different when Dired is in the Emacs repository?  The
Emacs repository is a public one, so anyone and everyone can get the
latest version from there and use it, if they want.

> I also think that discussion around this topic is distorted by a
> long-tailed distribution.  Most people probably have in mind the big
> emacs packages with dynamic developer communities and independent
> release cycles.  Org, Gnus, CEDET, and maybe a few others.  On the other
> hand, it seems to me that you have in mind (in addition to these) all
> the tiny packages that see very few commits and perhaps no new features
> (in the extreme case take kermit.el, which has seen no substantive
> changes since at least 1992, but still gets its copyright header
> lovingly updated every year).  I don’t know how to reconcile these
> viewpoints, or even if it’s necessary.  Just something to consider.

The suggestion was to move _all_ of them, except the few that are
needed for bootstrap, out of the Emacs repository.  Most of the
packages in that category are neither like Org nor like kermit.  They
are relatively small, but get quite a significant number of changes.

> Speaking for myself, my primary interest in emacs development is working
> on org-mode, and I heavily use org as well as third-party packages for
> python (elpy), clojure (cider), R (ESS), and a few other random things.
> I build emacs from git every month or so in order to pick up little
> quality of life improvements, like with-eval-afer-load, some of the
> subr-x stuff, seq.el, the overhauled package menu, etc.  But I have to
> make sure to keep a couple months’ worth of old emacsen around, in case
> my monthly pull happens to land on a buggy commit – one that causes
> regular segfaults, which I have had happen more than once.  I use emacs
> for work so I don’t have the luxury of just going without for a few days
> until the problem is fixed.  I could go back to a previous released
> version of course, but then my init.el breaks because it relies on new
> features since the release.

I don't see how the issue at hand can affect you, then.  Your usage is
very similar to mine, FWIW.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 05:42 AM, Eli Zaretskii wrote:

>> Imagine someone implements an awesome new feature for dired.  Emacs
>> users the world over are amazed by this, and fill their blogs, twitter,
>> etc. with the news.  If dired is an ELPA package, everyone who hears
>> this news can get the new feature in their emacs instantly by upgrading
>> their ELPA packages.  No need to wait N months for a new release of
>> emacs, or compile a non-release version of emacs from git.
>
> How is this different when Dired is in the Emacs repository?  The
> Emacs repository is a public one, so anyone and everyone can get the
> latest version from there and use it, if they want.

a) That's a more involved endeavor than installing a package from ELPA. 
And then you don't get the same conveniences, such as automatic updates.

b) There's a much higher probability that Dired depends on something 
only the current development version of Emacs has. ELPA packages declare 
their version requirements explicitly, and try not to break 
compatibility with earlier versions without sufficient reasons.

> The suggestion was to move _all_ of them, except the few that are
> needed for bootstrap, out of the Emacs repository.  Most of the
> packages in that category are neither like Org nor like kermit.  They
> are relatively small, but get quite a significant number of changes.

There were different suggestions, with different degrees between "let's 
move Org and Gnus out" and "let's move everything out".

New ELPA policy proposal (was: ELPA policy)

[Oleh Krehel]

Hi all,

In the previous thread, a few concerns were posted:

1. Having ELPA separate from Emacs requires internet access to install
extra packages, which might be undesirable.

2. Merging all ELPA commits into Emacs might be too noisy.

3. Waiting for new core features until Emacs release, as opposed to
obtaining them immediately from ELPA.

Here's my proposal to address these issues:

1. Only packages which are relatively independent are allowed to be
moved to ELPA. This is a rule that we adhere to right now, I just wanted
to re-state it. And it's important for the next point.

2. Each package is developed it its own Git repository, and is included
into the common ELPA directory as a Git submodule.  No actual code is
ever committed into the Emacs git repository, only the commit hashes,
when an ELPA package maintainer deems it appropriate to merge, e.g. the
package is assumed to work fine with the current Emacs git version, at
the moment of the merge. Then there will also be some script, like

    $ make elpa

that will update all packages with "git submodule update".  In my
opinion, this is very close the model of MELPA - a very successful Emacs
package archive with 2786 packages to GELPA's 130. The only difference
here is that the updates aren't automatic, and are instead up to the
maintainer, which should give more stability.

3. On Emacs release, we'll have emacs.tar (for people with low
bandwidth) and emacs-with-elpa.tar (the recommended download). The user
will have several options to configure the Emacs installation:

    a. Only core Emacs.

    b. Core Emacs with a separate offline ELPA repository. The user will
    be able to install from this offline ELPA repository at any time.
    This should be a nice default option, I think.

    c. Emacs with all ELPA packages installed, because why not?

4. Thanks to the version system, it will be possible to update these
"offline" ELPA packages with the current online version, just like it's
currently possible to have the GELPA version of a package to update to
the MELPA version. Additionally, with the feature of pinning a package
to an archive, we give the user the flexibility to pin ELPA packages to
the offline ELPA repository, instead of having them update automatically
from GNU ELPA.

regards,
Oleh

Re: New ELPA policy proposal (was: ELPA policy)

[Artur Malabarba]

2015-11-09 11:15 GMT+00:00 Oleh Krehel <ohwoeowho@gmail.com>:
> 1. Only packages which are relatively independent are allowed to be
> moved to ELPA. This is a rule that we adhere to right now, I just wanted
> to re-state it. And it's important for the next point.
>
> 2. Each package is developed it its own Git repository, and is included
> into the common ELPA directory as a Git submodule.  No actual code is
> ever committed into the Emacs git repository, only the commit hashes,
> when an ELPA package maintainer deems it appropriate to merge,

Two Things:

* The first and simplest issue here is that we would lose control over
the code on Gelpa.

We can still remove any package from Gelpa entirely, of course. But it
would be difficult for the Emacs maintainers to edit a package on
Gelpa that's doing something wrong. For instance, Stefan has edited
several of my packages in the past (which I'm grateful for), and he
wouldn't have been able to do that if the source wasn't on elpa.git.

But maybe that's just a cost that needs to be paid if we want Gelpa to scale.

* Another thing to remember is that we need to be able to trust that
code made available by Gelpa has been properly copyright assigned.

The current way we do that is to:
1. Only make available code that has been pushed to elpa.git.
2. We trust those with push access to only push code that's been
assigned (to the best of their knowledge).
3. We trust that those who submit code are not lying about its origins.

IIUC, your proposal is that new releases would be marked by editing a
file in the Gelpa repo and adding a hash of the relevant commit.
This sounds more or less ok:
2. The manual edit of a hash would still have to be done by people
with push access, so only they could ultimately make new code
available on Gelpa.
3. The act of a developer asking us to edit a hash could be considered
analogous to the act submitting a package (i.e., they are declaring
that all the new code in that commit has been properly assigned, just
like they originally declared so when first submitting the package).

> 3. On Emacs release, we'll have emacs.tar (for people with low
> bandwidth) and emacs-with-elpa.tar (the recommended download).

I'm perfectly in favor of this. And it's pretty much independent of
the points above. Of course, the inner workings of Gelpa affect how
the build-script might create this emacs-with-elpa.tar, but they don't
affect the feasibility or difficulty of this task.

So it's just a matter of someone writting the necessary
scripts/documentation for the next release.

Re: New ELPA policy proposal

[Oleh Krehel]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Two Things:
>
> * The first and simplest issue here is that we would lose control over
> the code on Gelpa.
>
> We can still remove any package from Gelpa entirely, of course. But it
> would be difficult for the Emacs maintainers to edit a package on
> Gelpa that's doing something wrong. For instance, Stefan has edited
> several of my packages in the past (which I'm grateful for), and he
> wouldn't have been able to do that if the source wasn't on elpa.git.
>
> But maybe that's just a cost that needs to be paid if we want Gelpa to scale.

This doesn't have to be so. GELPA is a git repository on
Savannah. Anyone with proper SSH keys can edit it.  What I propose is to
split GELPA into 130 git repositories on Savannah (or separate git
branches of one repository if that's more convenient).  All Emacs devs
will still have the same access to every package. But the commit log
won't be a mingled mess: each package will have its own commit log and
its own version - the latest version tag on the appropriate branch.
Then the main ELPA directory will hold the current version hashes for
each ELPA package. And the main Emacs repository will hold one ELPA hash
snapshot for each Emacs release.

> * Another thing to remember is that we need to be able to trust that
> code made available by Gelpa has been properly copyright assigned.
>
> The current way we do that is to:
> 1. Only make available code that has been pushed to elpa.git.
> 2. We trust those with push access to only push code that's been
> assigned (to the best of their knowledge).
> 3. We trust that those who submit code are not lying about its origins.
>
> IIUC, your proposal is that new releases would be marked by editing a
> file in the Gelpa repo and adding a hash of the relevant commit.
> This sounds more or less ok:
> 2. The manual edit of a hash would still have to be done by people
> with push access, so only they could ultimately make new code
> available on Gelpa.
> 3. The act of a developer asking us to edit a hash could be considered
> analogous to the act submitting a package (i.e., they are declaring
> that all the new code in that commit has been properly assigned, just
> like they originally declared so when first submitting the package).

I assume the process could have several tiers.

tier 1: I edit my package in a Git repository on my machine and accept
contributions with CA.

tier 2: I sync my Git repository with its Git repository on Savannah (a
    sub-directory of GELPA)

tier 3: The main directory of GELPA should always hold the hashes
     of all packages, the stable versions that are to be published to
     https://elpa.gnu.org/packages/. This information will be updated
     with each new release of any package.

tier 4: On release of a new Emacs version, a snapshot of GELPA hashes
    will be committed into Emacs. All code that corresponds to that
    snapshot will be bundled with emacs-with-elpa.tar.

Currently, tier 3 is done with ";; Version: ..." cookie. I propose that
we do it with git tags and hashes instead/additionally.
As for tier 4, the package maintainers take the responsibility to ensure
that the package hash included into Emacs works correctly around the time
when each feature freeze is announced.

The above system can solve an inconvenience that I've occasionally run
into with the current GELPA setup: the "git subtree merge" doesn't allow
me to easily pull anyone's changes done in GELPA back into my package.
In fact, at times I haven't noticed a change for a month, committed a
bunch on top of the version in GELPA, and then had to do a rebase and
resolve merge conflicts.

In the end, I'd like to setup a workflow where my package has two
identical public fronts: one on Github and one on Savannah. Pushing to
one should push to the other one.  And if I pull from both, I'll be
immediately notified if any other Emacs developer edited my package on
Savannah, and I'll just commit on top of these changes.

Here's a visual representation of the ELPA directory on Savannah that I
propose:

|-elpa/
|--.git/
|--ace-window/
|---./git/
|--ack/
|---.git/
...
|--.gitignore
|--package-list

Everyone that currently has access to ./elpa/ will automatically have
access to ./elpa/*.
Each sub-directory is its own git repository, and ./.gitignore will ignore
everything except ./package-list, and maybe some auxiliary scripts.

./package-list could have the following structure:

    ace-window: eef897e590c4ce63c28fd29ebff3c97aec8a69ae
    ack: 1487ae155ccc2a3099b8e0f20334a79fa8f5b9a8
    ...

In the end, ./package-list could simply be built automatically by a
shell command examining each repository for a version tag.  But it might
be good to have each new version release accompanied by a commit to
./package-list.

Re: New ELPA policy proposal

[Wolfgang Jenkner]

On Mon, Nov 09 2015, Artur Malabarba wrote:

> * The first and simplest issue here is that we would lose control over
> the code on Gelpa.

But when is control over elisp code not bundled in the emacs tarball
actually needed?  I'd say, only in two cases:

(1) code that is poised to be bundled in a short- or mid-term
perspective.

(2) code that is removed from the bundled code, until
a separate maintainer is found for it.

So, a policy of minimal control would have bundled code in emacs.git
(and nowhere else), while elpa would be merely a staging area for it.

Re: New ELPA policy proposal

[Dmitry Gutov]

On 11/09/2015 05:41 PM, Wolfgang Jenkner wrote:
> On Mon, Nov 09 2015, Artur Malabarba wrote:
>
>> * The first and simplest issue here is that we would lose control over
>> the code on Gelpa.
>
> But when is control over elisp code not bundled in the emacs tarball
> actually needed?

So far, the code that's been pushed to ELPA, has all been more or less 
reviewed.

AFAIK, when Stefan pushes new versions (if that still happens 
semi-manually), he can make sure that nothing precarious has sneaked 
into one of the packages.

Since GNU ELPA is the official repository, the Emacs developers have a 
responsibility over its contents. Not just individual developers.

Re: ELPA policy

[Eli Zaretskii]

> Cc: Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 05:51:16 +0200
> 
> On 11/09/2015 05:42 AM, Eli Zaretskii wrote:
> 
> >> Imagine someone implements an awesome new feature for dired.  Emacs
> >> users the world over are amazed by this, and fill their blogs, twitter,
> >> etc. with the news.  If dired is an ELPA package, everyone who hears
> >> this news can get the new feature in their emacs instantly by upgrading
> >> their ELPA packages.  No need to wait N months for a new release of
> >> emacs, or compile a non-release version of emacs from git.
> >
> > How is this different when Dired is in the Emacs repository?  The
> > Emacs repository is a public one, so anyone and everyone can get the
> > latest version from there and use it, if they want.
> 
> a) That's a more involved endeavor than installing a package from ELPA. 
> And then you don't get the same conveniences, such as automatic updates.

Nothing prevents us from making the same arrangements in the Emacs
repository as we have in ELPA, and then users could add the Emacs
repository to their list of sites that package.el knows about.

Besides, most core packages don't need any elaborate setup, you just
drop them in and restart Emacs.

> b) There's a much higher probability that Dired depends on something 
> only the current development version of Emacs has. ELPA packages declare 
> their version requirements explicitly, and try not to break 
> compatibility with earlier versions without sufficient reasons.

How will that change if we move them outside Emacs?  It's not like
there's some magic at ELPA that just putting a package there magically
resolves all these issues.  And if someone will have to work on that,
she can do that without moving the package aside.

> > The suggestion was to move _all_ of them, except the few that are
> > needed for bootstrap, out of the Emacs repository.  Most of the
> > packages in that category are neither like Org nor like kermit.  They
> > are relatively small, but get quite a significant number of changes.
> 
> There were different suggestions, with different degrees between "let's 
> move Org and Gnus out" and "let's move everything out".

I didn't see any.

Re: New ELPA policy proposal (was: ELPA policy)

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 09 Nov 2015 12:15:29 +0100
> 
> 3. On Emacs release, we'll have emacs.tar (for people with low
> bandwidth) and emacs-with-elpa.tar (the recommended download). The user
> will have several options to configure the Emacs installation:
> 
>     a. Only core Emacs.
> 
>     b. Core Emacs with a separate offline ELPA repository. The user will
>     be able to install from this offline ELPA repository at any time.
>     This should be a nice default option, I think.
> 
>     c. Emacs with all ELPA packages installed, because why not?

Why not have only one distribution, the full one?

Re: New ELPA policy proposal

[Wolfgang Jenkner]

On Mon, Nov 09 2015, Dmitry Gutov wrote:

> Since GNU ELPA is the official repository, the Emacs developers have
> a responsibility over its contents. Not just individual developers.

Sure, but how does this contradict what I said?

I suggested a `policy of minimal control' where elpa remains an official
repository, but with transient content.

I'm aware that this is different from elpa's current purpose.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:05 PM, Eli Zaretskii wrote:

> Nothing prevents us from making the same arrangements in the Emacs
> repository as we have in ELPA, and then users could add the Emacs
> repository to their list of sites that package.el knows about.

Create yet another package archive? With a separate publishing mechanism?

> Besides, most core packages don't need any elaborate setup, you just
> drop them in and restart Emacs.

"Just drop them in" doesn't address the issue of package updates, not in 
the slightest.

> How will that change if we move them outside Emacs?  It's not like
> there's some magic at ELPA that just putting a package there magically
> resolves all these issues.  And if someone will have to work on that,
> she can do that without moving the package aside.

They will be forced to declare the required Emacs version, for one. And 
abide by it.

But yes, that's also possible to do inside the Emacs repo.

>> There were different suggestions, with different degrees between "let's
>> move Org and Gnus out" and "let's move everything out".
>
> I didn't see any.

All right, then. Let me make one right now: let's move out Org, Gnus and 
CEDET. And consider doing that with any other big codebase that's also 
maintained separately (Calc comes to mind).

There have been objections to doing that for each of these packages, but 
there you go. Suggestion made.

Re: ELPA policy

[Eli Zaretskii]

> Cc: aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 18:15:28 +0200
> 
> On 11/09/2015 06:05 PM, Eli Zaretskii wrote:
> 
> > Nothing prevents us from making the same arrangements in the Emacs
> > repository as we have in ELPA, and then users could add the Emacs
> > repository to their list of sites that package.el knows about.
> 
> Create yet another package archive? With a separate publishing mechanism?

Why not?  There are several already, no?

> > Besides, most core packages don't need any elaborate setup, you just
> > drop them in and restart Emacs.
> 
> "Just drop them in" doesn't address the issue of package updates, not in 
> the slightest.

It does, for most of the core.  That's what you do each time you say
"git pull", I believe.

> >> There were different suggestions, with different degrees between "let's
> >> move Org and Gnus out" and "let's move everything out".
> >
> > I didn't see any.
> 
> All right, then. Let me make one right now: let's move out Org, Gnus and 
> CEDET. And consider doing that with any other big codebase that's also 
> maintained separately (Calc comes to mind).
> 
> There have been objections to doing that for each of these packages, but 
> there you go. Suggestion made.

It was made before -- that's the "let's move Org and Gnus out"
variant, to which I said I could easily agree.  And then there was the
"move everything" one, to which I object for the reasons stated.  What
I meant was that there was no 3rd variant, AFAIR.

Re: New ELPA policy proposal

[Artur Malabarba]

Ok, I had misunderstood you. I thought you were suggesting we do some
automated pulling from arbitrary remote repos. But I see you still want
to keep the code in savannah repos, which is indeed more reasonable.

However, I have no idea of the logistics involved in setting up several
(or several dozen?) savannah repos. So it would be nice if one of the
big shots could provide feedback.

Oleh Krehel <ohwoeowho@gmail.com> writes:
> What I propose is to split GELPA into 130 git repositories on
> Savannah.

Let's slow down a bit. :-)
Plenty of packages require very little maintenance are doing just fine
as a part of the repository itself. It would be nothing but a pain to
spread all of these small packages into their own repos.

Other than that, I'm all on board. Giving packages _the possibility_ of
being a savannah repo (which are then added as submodules of elpa.git)
sounds right and good for me. And I know that many packages would
cherish this possibility.

If that turns out to work well, we can maybe make it so that the only
way of having "external" packages (dropping the current two methods). So
every package will either be directly in elpa.git or a savannah
submodule.

> All Emacs devs will still have the same access to every package. But
> the commit log won't be a mingled mess: each package will have its own
> commit log and its own version - the latest version tag on the
> appropriate branch.

All good. (Again, as long as it's a possibility, not a demand. Most
packages barely make a dent on the commit log.)

Re: New ELPA policy proposal

[Artur Malabarba]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> So, a policy of minimal control would have bundled code in emacs.git
> (and nowhere else), while elpa would be merely a staging area for it.

I'm not sure if I understand you correctly. But Gelpa code will always
be available online, regardless of whether we bundle it into Emacs
releases or not.

So Gelpa is not a staging area, and the code that goes into it does need
to be under our control.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:20 PM, Eli Zaretskii wrote:

>> Create yet another package archive? With a separate publishing mechanism?
>
> Why not?  There are several already, no?

So far, it seems we're short of manpower to manage even the current one. 
To manage the infrastructure, improve the scripts, etc.

> It does, for most of the core.  That's what you do each time you say
> "git pull", I believe.

I cannot "git pull" a new version of Dired into my stable installation 
of Emacs 24.5.

> It was made before -- that's the "let's move Org and Gnus out"
> variant, to which I said I could easily agree.  And then there was the
> "move everything" one, to which I object for the reasons stated.  What
> I meant was that there was no 3rd variant, AFAIR.

Well, I enumerated 4 packages instead of 2, didn't I? :)

I agree that "move everything" is a non-starter (or very hard, at 
least). So it seems more like a red herring, there's no real use arguing 
against it.

Re: ELPA policy

[Artur Malabarba]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Nothing prevents us from making the same arrangements in the Emacs
>> > repository as we have in ELPA, and then users could add the Emacs
>> > repository to their list of sites that package.el knows about.
>> 
>> Create yet another package archive? With a separate publishing mechanism?
>
> Why not?  There are several already, no?

This sounds already quite similar to the system that Fabian implemented
a couple months ago (which I hear isn't quite perfect yet, but should be
almost there).

With this system, the daily script that Gelpa runs would copy some
packages straight from Emacs core. This way you could very well
configure Gelpa to always offer the lastest Dired (or maybe only when we
bump a version number, but that's a detail).

Re: ELPA policy

[Eli Zaretskii]

> Cc: aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 18:26:58 +0200
> 
> On 11/09/2015 06:20 PM, Eli Zaretskii wrote:
> 
> >> Create yet another package archive? With a separate publishing mechanism?
> >
> > Why not?  There are several already, no?
> 
> So far, it seems we're short of manpower to manage even the current one. 
> To manage the infrastructure, improve the scripts, etc.

Is it really such a significant effort?  I wasn't aware of that.

> > It does, for most of the core.  That's what you do each time you say
> > "git pull", I believe.
> 
> I cannot "git pull" a new version of Dired into my stable installation 
> of Emacs 24.5.

Of course you can: just copy the file over.

> > It was made before -- that's the "let's move Org and Gnus out"
> > variant, to which I said I could easily agree.  And then there was the
> > "move everything" one, to which I object for the reasons stated.  What
> > I meant was that there was no 3rd variant, AFAIR.
> 
> Well, I enumerated 4 packages instead of 2, didn't I? :)

We could probably find a few more.  MH-E, for example.  (Of course,
their developers will have to agree to move.)

> I agree that "move everything" is a non-starter (or very hard, at 
> least). So it seems more like a red herring, there's no real use arguing 
> against it.

Agreed.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:44 PM, Eli Zaretskii wrote:

>> So far, it seems we're short of manpower to manage even the current one.
>> To manage the infrastructure, improve the scripts, etc.
>
> Is it really such a significant effort?  I wasn't aware of that.

There are things to be done, and just a couple months ago nobody was 
doing them. I see some movement now, e.g. the last commit by Thomas 
Fitzsimmons.

>> I cannot "git pull" a new version of Dired into my stable installation
>> of Emacs 24.5.
>
> Of course you can: just copy the file over.

Automatic update is a really handy feature when you have 50+ packages 
installed. See some user statistics at 
https://www.reddit.com/r/emacs/comments/3s4k03/how_many_packages_do_you_use/

> We could probably find a few more.  MH-E, for example.  (Of course,
> their developers will have to agree to move.)

Yeah, I bet the majority of our users are unaware of what MH-E is.

Re: ELPA policy

[Achim Gratz]

Eli Zaretskii writes:
> The suggestion was to move _all_ of them, except the few that are
> needed for bootstrap, out of the Emacs repository.

You keep arguing about things that realistically aren't even on the
table yet (I don't want to prevent that discussion, mind you).  I said
it would be possible to do this, given some not-yet-existing
infrastructure and the only reason I even mentioned it was because
that's the other end of the spectrum from moving everything that looks
halfway useful into core.  Even if that infrastructure develops (which I
have serious doubts about given how this discussion goes), nobody in
their right mind would move all packages big-bang style to ELPA.
There'd certainly be a transition period with some pilot packages, I'd
probably pick Org, CEDET and Tramp based on the list traffic.  You'd
then see how that works and make some fixes that will certainly be
necessary before you get to the point to move more packages over to
ELPA.  This being Emacs, for better or worse, I'd expect that at this
point we'd be looking at _years_ before the extreme end you seem to be
so worried about is even within reach.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: ELPA policy

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Mon, 09 Nov 2015 19:22:50 +0100
> 
> Eli Zaretskii writes:
> > The suggestion was to move _all_ of them, except the few that are
> > needed for bootstrap, out of the Emacs repository.
> 
> You keep arguing about things that realistically aren't even on the
> table yet (I don't want to prevent that discussion, mind you).

If that's not on the table, then please tell what in your opinion _is_
on the table.  You did want to propose something practical that could
be done soon, right?  Because this discussion was about what we want
to do from now on with ELPA, not what we might want considering in
some distant future that might never come.

So please, let's be practical and consider alternatives that
realistically can be taken soon.  Like tomorrow or the next month.
Okay?

> There'd certainly be a transition period with some pilot packages, I'd
> probably pick Org, CEDET and Tramp based on the list traffic.

Moving out a few large packages that are developed outside Emacs
anyway is a no-brainer, provided that their developers agree.  I
already said I'll probably agree, and no one else objected.  The only
question is what is the opinion of the respective developers: we
cannot move the packages if they disagree.

Is there some other alternative on the table?  If not, we can finally
agree about something, I think.

Thanks.

Re: ELPA policy

[David Engster]

Eli Zaretskii writes:
>> From: Achim Gratz <Stromeko@nexgo.de>
>> Date: Mon, 09 Nov 2015 19:22:50 +0100

>> There'd certainly be a transition period with some pilot packages, I'd
>> probably pick Org, CEDET and Tramp based on the list traffic.
>
> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.  I
> already said I'll probably agree, and no one else objected.  The only
> question is what is the opinion of the respective developers: we
> cannot move the packages if they disagree.

As long as Emacs core cannot depend on packages that are bundled from
ELPA, I would definitely not agree to move CEDET to ELPA.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Mon, 09 Nov 2015 19:58:29 +0100
> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
> 
> As long as Emacs core cannot depend on packages that are bundled from
> ELPA, I would definitely not agree to move CEDET to ELPA.

Of course.  I don't think we should move anything before such a
dependency is possible.

Re: ELPA policy

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Date: Mon, 09 Nov 2015 19:58:29 +0100
>> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
>> 
>> As long as Emacs core cannot depend on packages that are bundled from
>> ELPA, I would definitely not agree to move CEDET to ELPA.
>
> Of course.  I don't think we should move anything before such a
> dependency is possible.

But if you have such dependencies, what's the point? IIRC, Stefan
decidedly did not want such dependencies, since the whole point was to
add those bundles only when creating releases.

-David

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> It was made before -- that's the "let's move Org and Gnus out" variant, to
> which I said I could easily agree. And then there was the "move everything"
> one, to which I object for the reasons stated. What I meant was that there
> was no 3rd variant, AFAIR.

Wow, a lot of discussion about ELPA policy, this is great. We definitely have
an opportunity here to bring clarity to an area that is on people's minds.

I agree with a lot of what I read, although it was a too spread out for me to
add specific quotes here. Let me just summarize a possible path forward:

 1. Things that are maintained by the core Emacs developers should be in core,
    and in the Emacs.git repository. This makes it easy for them to access and
    build, search history, read emacs-diffs, etc.

 2. Things that are maintained outside of Emacs, but contributed back for
    inclusion, should be in ELPA, and in the Elpa.git repository. This
    includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
    so let's start with the big ones).

 3. There should be a defined subset of packages that get copied from ELPA
    into the Emacs tarball during release, and an easy way to manage this list
    for the core developers. That way, certain packages like seq.el and
    stream.el can feel like "core" packages to users, when they are really
    "external" packages from the point of view of the core developers.

 4. Everything else in ELPA is Internet-installation based, as it is today.

I would very much like for ELPA to be a way to:

  - give the core developers a smaller surface area to focus on;

  - allow ELPA package maintainers to "have their package in Emacs", while
    retaining nimble in their own development process and in delivering
    updates to users;

  - allow the core maintainers to decide what exactly is going into "Emacs":
    For example, I wouldn't want to pull Org releases into the distribution
    from a submodule; I really do want a version of that source code to be in
    ELPA, so we can separately patch if there are last minute problems.

ELPA should thus benefit core developers by reducing load, and benefit package
maintainers by being an easier platform for their contributions and their
users. If it causes extra work for anyone, that's something I'd want to
change.

There are three actions I'd like to take from here:

  a. That we clearly document an ELPA policy we all agree on;

  b. That we move "external" packages from core into ELPA, starting with the
     really big ones;

  c. That we assess any points of friction after doing so, and adjust as
     necessary.

John

Re: ELPA policy

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.  I
> already said I'll probably agree, and no one else objected.  The only
> question is what is the opinion of the respective developers: we
> cannot move the packages if they disagree.

I'm pretty sure Org would be happy to be distributed this way.  In fact,
we have got the 8.3 release now.  It really *must* be in 25.1, but I think
it would be fine to try out Fabian’s system to allow it to be upgraded
later.  Otherwise, I will hopefully be able to merge it soon the
"copy-paste" way.

Rasmus


-- 
However beautiful the theory, you should occasionally look at the evidence

Re: ELPA policy

[Achim Gratz]

Eli Zaretskii writes:
> If that's not on the table, then please tell what in your opinion _is_
> on the table.  You did want to propose something practical that could
> be done soon, right?  Because this discussion was about what we want
> to do from now on with ELPA, not what we might want considering in
> some distant future that might never come.

I did want to propose something that could become a vision for some time
in the foreseeable, yet somewhat distant future that we can then take
steps towards.  ELPA is foremost for _users_ and any decisions taken
should put the users first and developers second.  As a user I don't
have the luxury of chosing which Emacs version I get to use and what
packages are already installed on the system (indeed in my day job that
changes when I switch between project workspaces).  Wouldn't it be
splendid if ELPA helped me making those differences fade into the
background, if not disappear entirely?

> So please, let's be practical and consider alternatives that
> realistically can be taken soon.  Like tomorrow or the next month.
> Okay?

No, not okay.  There's still no consensus of even roughly where we want
to end up, so starting a random walk to find out who's feeling well or
not with whichever direction is certainly not my idea of progress.

> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.

Again, it's not.  It simply doesn't work at this moment, not for Emacs,
not for the package developers and not for the users.  That doesn't have
anything to do with the size of those packages best I can tell.  To
change that (without moving the sources anywhere, just to not go down
that rabbit hole again) we'd need to devise a way to have an Emacs core
package whose autoload definitions can be separated and that can be
overridden (maybe only with a newer version) on the installation level
or by the user.  That would be the first step before any others can
follow, provided you even want to go that direction.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Mon, 09 Nov 2015 20:35:04 +0100
> 
> Eli Zaretskii writes:
> >> From: David Engster <deng@randomsample.de>
> >> Date: Mon, 09 Nov 2015 19:58:29 +0100
> >> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
> >> 
> >> As long as Emacs core cannot depend on packages that are bundled from
> >> ELPA, I would definitely not agree to move CEDET to ELPA.
> >
> > Of course.  I don't think we should move anything before such a
> > dependency is possible.
> 
> But if you have such dependencies, what's the point? IIRC, Stefan
> decidedly did not want such dependencies, since the whole point was to
> add those bundles only when creating releases.

I'm not pushing for the move, I just said that if the technical
problems are successfully resolved, I won't object.

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

They could (and probably should) effectively be in the build environment
as well.  In other words, I would try to not pull them into the release
tarball at the last possible moment.  One thing that would make
difficult is to test interactions between such "external core" packages
and here I have to agree with Eli that this has potential for
degradation of quality.

>   - allow the core maintainers to decide what exactly is going into "Emacs":
>     For example, I wouldn't want to pull Org releases into the distribution
>     from a submodule; I really do want a version of that source code to be in
>     ELPA, so we can separately patch if there are last minute problems.

You'd do that with branches if it really becomes necessary.  I don't
really see why submodules could not be used, except for the extra rope
they give you to hang yourself with.  Any other solution is going to
have the same basic complexity beneath and the potential to not work on
some platform or other.

> ELPA should thus benefit core developers by reducing load, and benefit package
> maintainers by being an easier platform for their contributions and their
> users. If it causes extra work for anyone, that's something I'd want to
> change.

I maintain that ELPA should benefit Emacs users first.

>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;

...after the necessary changes to Emacs' build system and package.el are
architected and implemented.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: ELPA policy

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> They could (and probably should) effectively be in the build environment as
> well. In other words, I would try to not pull them into the release tarball
> at the last possible moment. One thing that would make difficult is to test
> interactions between such "external core" packages and here I have to agree
> with Eli that this has potential for degradation of quality.

Could this be solved by branching ELPA leading up to a release -- this being
the "frozen ELPA" that we do integration against before cutting the tarball --
while cherry-picking fixes in from master? I think the way we proceed here
would be similar to what we do for Emacs itself.

> You'd do that with branches if it really becomes necessary. I don't really
> see why submodules could not be used, except for the extra rope they give
> you to hang yourself with. Any other solution is going to have the same
> basic complexity beneath and the potential to not work on some platform or
> other.

A submodule doesn't allow us to have "Org + some-patch-not-yet-in-Org".

> I maintain that ELPA should benefit Emacs users first.

I'm intrigued; could you clarify this a bit more? I saw what you wrote in
response to Eli, but that didn't really help. Specifically, do you have a
scenario in mind that is better for users and worse for developers?

>> b. That we move "external" packages from core into ELPA, starting with the
>> really big ones;

> ...after the necessary changes to Emacs' build system and package.el are
> architected and implemented.

Agreed.

John

Re: New ELPA policy proposal

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We must make sure that the principal Emacs developers can edit the code
in GELPA.

  > I shipped to Paris 1 baby GNU and 200 stickers of each type.

Thanks for telling me.  That means I can take a smaller quantity with
me from here.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I agree with a lot of what I read, although it was a too spread out for me to
  > add specific quotes here. Let me just summarize a possible path forward:

This plan seems good, overall, But in the specific case of stream.el
and seq.el, shouldn't they be maintained in the core?  I don't see
stream.el in the sources I fetched, but seq.el is small, in emacs-lisp,
and it says Maintainer: emacs-devel.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> This plan seems good, overall, But in the specific case of stream.el and
> seq.el, shouldn't they be maintained in the core? I don't see stream.el in
> the sources I fetched, but seq.el is small, in emacs-lisp, and it says
> Maintainer: emacs-devel.

If that's the case, then yes, it should be in core. I'm personally a bit
surprised that a library of this nature is in ELPA, unless it was being
provided and actively maintained by a non-core developer.

John

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On 9 Nov 2015 10:06 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
> >>>>> Richard Stallman <rms@gnu.org> writes:
>
> > This plan seems good, overall, But in the specific case of stream.el and
> > seq.el, shouldn't they be maintained in the core? I don't see stream.el
in
> > the sources I fetched, but seq.el is small, in emacs-lisp, and it says
> > Maintainer: emacs-devel.
>
> If that's the case, then yes, it should be in core. I'm personally a bit
> surprised that a library of this nature is in ELPA, unless it was being
> provided and actively maintained by a non-core developer.

Seq.el was put on Gelpa so that it can be used by emacsen < 25. I'm very
grateful for that as I use it as a dependency in quite a few of my packages
already.

[-- Attachment #2: Type: text/html, Size: 1009 bytes --]

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

I only added seq.el to ELPA for backward compatibility with Emacs 24.5.
The development of seq.el (and map.el for that matter) happen only in
Emacs trunk.

Moving it out of lisp/emacs-lisp/, and only having it copied in the
release tarballs would be a step backward IMHO, it would make it hard to
use it in core packages, and would basically mean demoting the libraries
to external ones while I'm really working on seq.el to provide a good,
consistent and comprehensive *built-in* sequence library to
Emacs-Lisp. That's also why all the functions are documented in the
Sequences chapter of the Elisp documentation.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Richard Stallman <rms@gnu.org> writes:

> This plan seems good, overall, But in the specific case of stream.el
> and seq.el, shouldn't they be maintained in the core?  I don't see
> stream.el in the sources I fetched, but seq.el is small, in emacs-lisp,
> and it says Maintainer: emacs-devel.

stream.el is currently in ELPA, which can make some sense, but regarding
seq.el, I think it should really stay in the core.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

> If that's the case, then yes, it should be in core. I'm personally a bit
> surprised that a library of this nature is in ELPA,

In the case of seq.el, it's for backward-compatibility.  It made it
possible for instance for CIDER devs to start using it while Emacs 25 is
still in progress.

> unless it was being
> provided and actively maintained by a non-core developer.

IIRC, Stefan thought that having stream.el in the ELPA meant more sense
at least until people start using it more.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Aaron Ecay]

Hi John,

Thanks for setting this out very concretely.

2015ko azaroak 9an, John Wiegley-ek idatzi zuen:
> 
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> It was made before -- that's the "let's move Org and Gnus out" variant, to
>> which I said I could easily agree. And then there was the "move everything"
>> one, to which I object for the reasons stated. What I meant was that there
>> was no 3rd variant, AFAIR.
> 
> Wow, a lot of discussion about ELPA policy, this is great. We definitely have
> an opportunity here to bring clarity to an area that is on people's minds.
> 
> I agree with a lot of what I read, although it was a too spread out for me to
> add specific quotes here. Let me just summarize a possible path forward:
> 
>  1. Things that are maintained by the core Emacs developers should be in core,
>     and in the Emacs.git repository. This makes it easy for them to access and
>     build, search history, read emacs-diffs, etc.
> 
>  2. Things that are maintained outside of Emacs, but contributed back for
>     inclusion, should be in ELPA, and in the Elpa.git repository. This
>     includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
>     so let's start with the big ones).
> 
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.
> 
>  4. Everything else in ELPA is Internet-installation based, as it is
>     today.

The one thing that I see missing from this list is a way to take
packages that are developed in emacs.git and distribute them on an
ELPA.  (Either GNU ELPA, or a (g)new ELPA as was suggested by Eli.)
This would be the inverse of your 3: packages that feel like core
packages to emacs devs, but feel like external packages to users (at
least in the sense of receiving updates outside of the emacs release
cycle).

I’m not sure if that omission was intentional or not.  I hope it wasn’t:
this model of distribution has been chosen for several important
“low-level” packages, like seq, cl-lib, and let-alist.

If this addition is desired then:

[...]

> 
> There are three actions I'd like to take from here:
> 
>   a. That we clearly document an ELPA policy we all agree on;
> 
>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;
> 
>   c. That we assess any points of friction after doing so, and adjust as
>      necessary.

This list needs an additional item, namely the development of the
script that publishes elisp from emacs.git to an ELPA.  I think people
have said that Fabian is already working on this.

Thanks,

-- 
Aaron Ecay

Re: ELPA policy

[John Wiegley]

>>>>> Aaron Ecay <aaronecay@gmail.com> writes:

> I’m not sure if that omission was intentional or not. I hope it wasn’t: this
> model of distribution has been chosen for several important “low-level”
> packages, like seq, cl-lib, and let-alist.

[...]

> This list needs an additional item, namely the development of the script
> that publishes elisp from emacs.git to an ELPA. I think people have said
> that Fabian is already working on this.

It was an unintentional omission, and I agree that we should support that
usage pattern as well. So let it be 4 modes of looking at "Emacs packages",
then.

John

Re: New ELPA policy proposal

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Date: Mon, 09 Nov 2015 12:15:29 +0100
>> 
>> 3. On Emacs release, we'll have emacs.tar (for people with low
>> bandwidth) and emacs-with-elpa.tar (the recommended download). The user
>> will have several options to configure the Emacs installation:
>> 
>>     a. Only core Emacs.
>> 
>>     b. Core Emacs with a separate offline ELPA repository. The user will
>>     be able to install from this offline ELPA repository at any time.
>>     This should be a nice default option, I think.
>> 
>>     c. Emacs with all ELPA packages installed, because why not?
>
> Why not have only one distribution, the full one?

If fine with it, if that's what we agree upon. Some people might still
care about bandwidth. I don't. But we do offer emacs-24.5.tar.gz and
emacs-24.5.tar.xz variants currently, I assume it's for the purposes of
bandwidth.

Re: New ELPA policy proposal

[John Wiegley]

>>>>> Oleh Krehel <ohwoeowho@gmail.com> writes:

>> Why not have only one distribution, the full one?

> If fine with it, if that's what we agree upon. Some people might still care
> about bandwidth. I don't. But we do offer emacs-24.5.tar.gz and
> emacs-24.5.tar.xz variants currently, I assume it's for the purposes of
> bandwidth.

I'd be fine with just one "full distribution" too, unless there are other
strong objections. It ought to be a difference of tens of megabytes at most.

John

Re: New ELPA policy proposal

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 10 Nov 2015 06:24:15 -0800
> 
> >>>>> Oleh Krehel <ohwoeowho@gmail.com> writes:
> 
> >> Why not have only one distribution, the full one?
> 
> > If fine with it, if that's what we agree upon. Some people might still care
> > about bandwidth. I don't. But we do offer emacs-24.5.tar.gz and
> > emacs-24.5.tar.xz variants currently, I assume it's for the purposes of
> > bandwidth.
> 
> I'd be fine with just one "full distribution" too, unless there are other
> strong objections. It ought to be a difference of tens of megabytes at most.

emacs-24.5.tar.xz is 38MB.  IMO, that's not "large" by any measure
nowadays.

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> It was made before -- that's the "let's move Org and Gnus out" variant, to
>> which I said I could easily agree. And then there was the "move everything"
>> one, to which I object for the reasons stated. What I meant was that there
>> was no 3rd variant, AFAIR.
>
> Wow, a lot of discussion about ELPA policy, this is great. We definitely have
> an opportunity here to bring clarity to an area that is on people's minds.
>
> I agree with a lot of what I read, although it was a too spread out for me to
> add specific quotes here. Let me just summarize a possible path forward:
>
>  1. Things that are maintained by the core Emacs developers should be in core,
>     and in the Emacs.git repository. This makes it easy for them to access and
>     build, search history, read emacs-diffs, etc.
>
>  2. Things that are maintained outside of Emacs, but contributed back for
>     inclusion, should be in ELPA, and in the Elpa.git repository. This
>     includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
>     so let's start with the big ones).
>
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

ELPA packages that other core code depends on (like CEDET; xref uses it
- called "core ELPA packages" hereinafter) have to be in every
developer's build environment everyday; the other core code has to see
the current version of the package, and it has to be the same for every
developer.

If core ELPA packages are in the ELPA git repository, you would copy
some subtrees of the ELPA git workspace into the Emacs git workspace.
People have looked into doing this, but it gets messy and confusing.
"git fetch" is dealing with two upstreams for one workspace. There has
been talk about git submodules, etc, but nothing concrete.

The alternative, which is being implemented by the :core external
package feature in Gnu ELPA, and has been shown to work (but has some
details to work out), is to keep core ELPA packages in the emacs
git repository. Then they are part of the developer's build environment
using the current git workflow, and they are included in both the emacs
tarball and the ELPA package server with the current release workflows.


There are two rationales for moving a package to ELPA:

1) Allow a release schedule independent of the core Emacs release
   schedule.

2) Reduce the size of the Emacs git workspace, which simplifies managing
   it.

But a _core_ ELPA package must remain in the Emacs git workspace, so the
only rationale for moving a core package to ELPA is for an independent
release schedule.

> There are three actions I'd like to take from here:
>
>   a. That we clearly document an ELPA policy we all agree on;

In admin/notes/elpa ?

>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;

It's not clear that "really big" implies "independent release schedule".
"externally managed" implies that.

"not used by other core code" is also a good reason to move a package to
ELPA. 

>   c. That we assess any points of friction after doing so, and adjust as
>      necessary.

If we use the approach of keeping core ELPA packages in the Emacs git
repository, there is _zero_ friction for the current core Emacs
developers; the only change is in the ELPA config scripts. 

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Seq.el was put on Gelpa so that it can be used by emacsen < 25.

It is fine to have it there, but this suggests we should maintain
it inside Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> ELPA packages that other core code depends on (like CEDET; xref uses it -
> called "core ELPA packages" hereinafter) have to be in every developer's
> build environment everyday; the other core code has to see the current
> version of the package, and it has to be the same for every developer.
> 
> If core ELPA packages are in the ELPA git repository, you would copy some
> subtrees of the ELPA git workspace into the Emacs git workspace. People have
> looked into doing this, but it gets messy and confusing. "git fetch" is
> dealing with two upstreams for one workspace. There has been talk about git
> submodules, etc, but nothing concrete.

Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
This allows us to expressly state which "version" of Elpa is expected to work
with the current Emacs.git.

Large packages like CEDET should move outside of Emacs.git and into Elpa.git.
If xref.el depends on CEDET, it would move to Elpa.git as well.

> If we use the approach of keeping core ELPA packages in the Emacs git
> repository, there is _zero_ friction for the current core Emacs developers;
> the only change is in the ELPA config scripts.

I do not want this code replication. We will promote ELPA as an active
development space alongside core Emacs. If we use a submodule, and improve our
tooling slightly, it should become a seamless thing.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:28 PM, John Wiegley wrote:

> If xref.el depends on CEDET, it would move to Elpa.git as well.

I can cut out xref.el's dependency on CEDET easily enough.

Moving it into ELPA would be kinda problematic, because guess what M-. 
and M-, and bound to by default now.

Re: ELPA policy

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> I can cut out xref.el's dependency on CEDET easily enough.

Great, let's do that. I'd prefer to have the functionality of xref.el in core.

Thanks,
  John

Re: ELPA policy

[David Engster]

Dmitry Gutov writes:
> On 11/10/2015 08:28 PM, John Wiegley wrote:
>
>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
> I can cut out xref.el's dependency on CEDET easily enough.

Which illustrates nicely the downside of moving packages to ELPA.

-David

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>> ELPA packages that other core code depends on (like CEDET; xref uses it -
>> called "core ELPA packages" hereinafter) have to be in every developer's
>> build environment everyday; the other core code has to see the current
>> version of the package, and it has to be the same for every developer.
>> 
>> If core ELPA packages are in the ELPA git repository, you would copy some
>> subtrees of the ELPA git workspace into the Emacs git workspace. People have
>> looked into doing this, but it gets messy and confusing. "git fetch" is
>> dealing with two upstreams for one workspace. There has been talk about git
>> submodules, etc, but nothing concrete.
>
> Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
> This allows us to expressly state which "version" of Elpa is expected to work
> with the current Emacs.git.

Since ELPA comprises many packages, that simply cannot work.

> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.

Why?

> I do not want this code replication. We will promote ELPA as an active
> development space alongside core Emacs. If we use a submodule, and improve our
> tooling slightly, it should become a seamless thing.

From my experience, git submodules are never seamless to work with.

-David

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Great, let's do that. I'd prefer to have the functionality of xref.el in core.

So that means xref won't support CEDET out of the box? What is gained by
that, exactly?

-David

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  aaronecay@gmail.com,  Stromeko@nexgo.de,  emacs-devel@gnu.org,  Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 10:28:26 -0800
> 
> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.

"Should" based on what? just the fact that it's large?  I think we
should decide this stuff on a case by case basis.  For example:

> If xref.el depends on CEDET, it would move to Elpa.git as well.

IMO, the exact opposite: if there are core features that we want to be
in Emacs no matter what, and those features depend on a package which
could be a candidate to move to ELPA, that package should NOT move to
ELPA.

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> So that means xref won't support CEDET out of the box? What is gained by
> that, exactly?

It could always support it with customization hooks.

John

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

>> Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
>> This allows us to expressly state which "version" of Elpa is expected to work
>> with the current Emacs.git.

> Since ELPA comprises many packages, that simply cannot work.

Perhaps I should say, "The version of Elpa.git, a subset of which will appear
in the release tarball of this commit".

>> Large packages like CEDET should move outside of Emacs.git and into
>> Elpa.git.

> Why?

There will never be 100% agreement on whether they should be in ELPA, or be in
Core, so I'm making the decision that they belong in ELPA. The big three that
will be moved first are: Gnus, Org and CEDET.

> From my experience, git submodules are never seamless to work with.

Even still, they work perfectly fine for this purpose.

John

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> Large packages like CEDET should move outside of Emacs.git and into
>> Elpa.git.
> "Should" based on what? just the fact that it's large? I think we should
> decide this stuff on a case by case basis. For example:

I'm surprised you say this, Eli, because in another thread you agree that
packages like this should be in Elpa, didn't you?

>> If xref.el depends on CEDET, it would move to Elpa.git as well.

> IMO, the exact opposite: if there are core features that we want to be in
> Emacs no matter what, and those features depend on a package which could be
> a candidate to move to ELPA, that package should NOT move to ELPA.

Core should provide functionality along the lines of a "standard library" and
a "standard environment", where having them in core is as much a statement
about consistency of interface, as it is about universal availability of the
functionality.

Since xref.el does not need to depend on CEDET, I don't see a reason why it
should, causing CEDET to remain in core.

John

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>
>> Why?
>
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA.

So you simply don't want to give any reason?

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> So you simply don't want to give any reason?

David, although I work by consensus as much as possible, I won't explain
everything I choose to do. My reason is that I think this will improve Emacs
development.

Part of my goal is to make the boundary between Emacs and ELPA thinner, so it
will actually be *easier* to move CEDET and friends back into core later, if
my reasoning is wrong.

John

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Tue, 10 Nov 2015 10:35:13 -0800
> Cc: aaronecay@gmail.com, Eli Zaretskii <eliz@gnu.org>, Stromeko@nexgo.de,
> 	Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
> > I can cut out xref.el's dependency on CEDET easily enough.
> 
> Great, let's do that.

Sounds like a step backward to me.  CEDET allows xref to be more
versatile and support more back-ends.

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,  aaronecay@gmail.com,  Eli Zaretskii <eliz@gnu.org>,  Stromeko@nexgo.de,  Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> Date: Tue, 10 Nov 2015 10:52:41 -0800
> 
> > Why?
> 
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA.

IMO, it's a mistake to move CEDET.

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> David Engster <deng@randomsample.de> writes:
>
>> So you simply don't want to give any reason?
>
> David, although I work by consensus as much as possible, I won't explain
> everything I choose to do. My reason is that I think this will improve Emacs
> development.

This is not about reaching a consensus. This is about you giving proper
reasons for such a big change.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: stephen_leake@stephe-leake.org,  aaronecay@gmail.com,  Stromeko@nexgo.de,  emacs-devel@gnu.org,  dgutov@yandex.ru
> Date: Tue, 10 Nov 2015 10:54:51 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Large packages like CEDET should move outside of Emacs.git and into
> >> Elpa.git.
> > "Should" based on what? just the fact that it's large? I think we should
> > decide this stuff on a case by case basis. For example:
> 
> I'm surprised you say this, Eli, because in another thread you agree that
> packages like this should be in Elpa, didn't you?

I said I _could_agree.  But not unconditionally.  We should discuss
each case.

> >> If xref.el depends on CEDET, it would move to Elpa.git as well.
> 
> > IMO, the exact opposite: if there are core features that we want to be in
> > Emacs no matter what, and those features depend on a package which could be
> > a candidate to move to ELPA, that package should NOT move to ELPA.
> 
> Core should provide functionality along the lines of a "standard library" and
> a "standard environment", where having them in core is as much a statement
> about consistency of interface, as it is about universal availability of the
> functionality.

Sorry, I don't understand what this means in practice.

> Since xref.el does not need to depend on CEDET, I don't see a reason why it
> should, causing CEDET to remain in core.

xref is as useful as its backends.  If you take away backends, it
becomes less useful, or supports less programming languages, or both.

Re: ELPA policy

[Eli Zaretskii]

How about if we leave this issue alone for a while?  Let's concentrate
on releasing Emacs 25.1 first, and return to this issue later.  Rome
wasn't built in one day, we don't have to rush.

WDYT?

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> How about if we leave this issue alone for a while? Let's concentrate on
> releasing Emacs 25.1 first, and return to this issue later. Rome wasn't
> built in one day, we don't have to rush.
> 
> WDYT?

Agreed. Let's table for now and revisit. If the suggested moves really trouble
core developers, we'll talk about it again before making a final decision.

John

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> This is not about reaching a consensus. This is about you giving proper
> reasons for such a big change.

To be clear, I would also put Eshell (though not pcomplete) into the category
of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
be in the release tarball.

It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
in ELPA today, and we were talking about moving it into core, I would have
just as much trouble justifying that move too. Perhaps this simply underscores
the fact that we don't have an agreed upon ELPA policy we can all refer to.

OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
will be, until it becomes clear, based on that document, what should go where,
and why. We'll defer discussions of package movement until we have that in
place.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:37 PM, David Engster wrote:

>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Which illustrates nicely the downside of moving packages to ELPA.

Nope. It will be beneficial to the code in Emacs, because xref is 
currently depending on a minor feature of CEDET (the 
semantic-symref-tool infrastructure), which could use a rewrite, to be 
available for a more general usage.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:44 PM, David Engster wrote:

> So that means xref won't support CEDET out of the box? What is gained by
> that, exactly?

It doesn't support CEDET even now. Nobody wrote the Semantic 
implementation of a xref backend, and it might be tool late now for the 
25.1 release. If CEDET were moved to ELPA, however, you can add that 
support at your leisure later.

And that's an orthogonal issue. xref has no need to support CEDET: CEDET 
should support xref (the support code will be in CEDET's tree).

Re: ELPA policy

[Eli Zaretskii]

> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>  Eli Zaretskii <eliz@gnu.org>, aaronecay@gmail.com, Stromeko@nexgo.de,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 21:57:25 +0200
> 
> On 11/10/2015 08:37 PM, David Engster wrote:
> 
> >> I can cut out xref.el's dependency on CEDET easily enough.
> >
> > Which illustrates nicely the downside of moving packages to ELPA.
> 
> Nope. It will be beneficial to the code in Emacs, because xref is 
> currently depending on a minor feature of CEDET (the 
> semantic-symref-tool infrastructure), which could use a rewrite, to be 
> available for a more general usage.

I think a point we should take from this is that the prerequisites (in
this case, the said rewrite) should be in place _before_ we seriously
consider such moves (pun intended ;-).

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> David Engster <deng@randomsample.de> writes:
>
>> This is not about reaching a consensus. This is about you giving proper
>> reasons for such a big change.
>
> To be clear, I would also put Eshell (though not pcomplete) into the category
> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
> be in the release tarball.
>
> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
> in ELPA today, and we were talking about moving it into core, I would have
> just as much trouble justifying that move too. Perhaps this simply underscores
> the fact that we don't have an agreed upon ELPA policy we can all refer to.

In my opinion, the main question is whether something provides
infrastructure for other packages to use. This is precisely what CEDET
tries to do. I wouldn't have much trouble with putting parts of CEDET in
ELPA, namely those parts that do not directly provide infrastructure,
like support for certain languages, project types, indexing tools, etc.

> OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
> will be, until it becomes clear, based on that document, what should go where,
> and why. We'll defer discussions of package movement until we have that in
> place.

It is still not clear to me what exactly is gained by moving core
packages to ELPA. In my opinion, packages like Org, Gnus or CEDET didn't
create significant problems for Emacs development in the past. On the
contrary, it made sure that those packages were adapted quickly when
larger changes in core were made. Those changes got synced back upstream
by people like Bastien, Katsumi and me. So in my opinion, you are trying
to fix something that is not broken.

What actually *does* create problems are packages that don't have an
active maintainer. The 'big three' however are not among those.

-David

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:49 PM, Eli Zaretskii wrote:

> IMO, the exact opposite: if there are core features that we want to be
> in Emacs no matter what, and those features depend on a package which
> could be a candidate to move to ELPA, that package should NOT move to
> ELPA.

xref only depends on a minor feature of CEDET (the code indexing tools 
integration), and I've done that in a quick-and-dirty way, as a proof of 
concept.

It would be better to have functionality like that in the core, 
independent of CEDET.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 09:21 PM, Eli Zaretskii wrote:

> xref is as useful as its backends.  If you take away backends, it
> becomes less useful, or supports less programming languages, or both.

None of the xref backends use Semantic. There are still only two of 
them: elisp and etags.

When CEDET grows a backend, it can just as well add it when 
semantic-mode is enabled. No need to have it in the core for that.

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
>> You'd do that with branches if it really becomes necessary. I don't really
>> see why submodules could not be used, except for the extra rope they give
>> you to hang yourself with. Any other solution is going to have the same
>> basic complexity beneath and the potential to not work on some platform or
>> other.
>
> A submodule doesn't allow us to have "Org + some-patch-not-yet-in-Org".

You'd branch on ELPA to something like emacs-25.0.50, patch away and
point the submodule there if you really can't wait for the patch to be
committed in Org upstream and mirrored back to ELPA.

>> I maintain that ELPA should benefit Emacs users first.
>
> I'm intrigued; could you clarify this a bit more? I saw what you wrote in
> response to Eli, but that didn't really help. Specifically, do you have a
> scenario in mind that is better for users and worse for developers?

It was a long day and I'm afraid I'd not be too coherent, so I'll punt
for now.  But in short, look at CPAN, CRAN and CTAN for NIH examples of
what ELPA might possibly become.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:01 PM, Eli Zaretskii wrote:

> I think a point we should take from this is that the prerequisites (in
> this case, the said rewrite) should be in place _before_ we seriously
> consider such moves (pun intended ;-).

The only prerequisite is to have this functionality (which is fairly 
self-contained) in the core. It would benefit from a rewrite (and moving 
away from CEDET would encourage that), but as the first step, we can 
just copy it and do some renaming.

However, like mentioned in the "project.el semantics" thread:

   I'm seriously considering to get away from it and always use Grep,
   because the other tools, while they can be faster, they're also
   entirely opaque to the user, can have outdated databases, non-indexed
   relevant directories, unsupported languages, and so on.

So this feature, which xref is depending on for the default 
xref-find-references implementation, requires certain amount of 
hand-holding from the user. And I don't know how to approach documenting 
it, and otherwise make it apparent to the user.

So someone else should probably start on taking care of that. Otherwise, 
I'd rather fall back on Grep, until such point that Emacs starts to 
manage the Global/id-utils/CScope datbases for the user, and learns to 
make sure that the database is up-to-date.

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> the main question is whether something provides infrastructure for other
> packages to use.

Sounds like a good sentence for an ELPA policy. :)

> It is still not clear to me what exactly is gained by moving core packages
> to ELPA.

Agility. What is appropriate. Knowing when a thing goes into core, and when in
ELPA.

Org is an application, it's not infrastructure; the same with Gnus. *Parts* of
Gnus might rightly be considered infrastructure, but the whole of Gnus just
doesn't belong there. Parts of CEDET probably do belong in Emacs core, but as
an application, I don't think the whole of it does.

Since we don't have an agreed upon basis by which to draw such lines, we're
both talking about what we feel is right. Let's agree to revisit after having
the ELPA discussion.

John

Re: ELPA policy

[Eli Zaretskii]

> Cc: deng@randomsample.de, stephen_leake@stephe-leake.org,
>  aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 22:19:46 +0200
> 
> So someone else should probably start on taking care of that. Otherwise, 
> I'd rather fall back on Grep, until such point that Emacs starts to 
> manage the Global/id-utils/CScope datbases for the user, and learns to 
> make sure that the database is up-to-date.

Grep doesn't scale well to large projects, IME.  You get too many
false positives.

Outdated databases are easy to avoid with the likes of cron jobs.
Yes, that's hand-holding, but when you have to quickly find stuff in a
project with 3 million lines of code and thousands of classes, there
really is no other alternative.

Re: ELPA policy

[David Engster]

Dmitry Gutov writes:
> On 11/10/2015 08:37 PM, David Engster wrote:
>
>>> I can cut out xref.el's dependency on CEDET easily enough.
>>
>> Which illustrates nicely the downside of moving packages to ELPA.
>
> Nope. It will be beneficial to the code in Emacs, because xref is
> currently depending on a minor feature of CEDET (the
> semantic-symref-tool infrastructure), which could use a rewrite, to be
> available for a more general usage.

As you know, we will drop our upstream repository. During the last
merge, I will also have to discuss with Eric what we can do to narrow
CEDET's scope. I agree with you that semantic-symref is probably one of
the packages that should be replaced with something more general. But as
long as it's there, I'd appreciate if you could leave whatever support
you have coded for it.

I'd also be happy to drop the completion GUI stuff, BTW. I've repeatedly
said that company should be in core for that.

-David

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:45 PM, David Engster wrote:

> As you know, we will drop our upstream repository. During the last
> merge, I will also have to discuss with Eric what we can do to narrow
> CEDET's scope.
> I agree with you that semantic-symref is probably one of
> the packages that should be replaced with something more general.

In the end, I would leave only the "backend" part of semantic-symref 
(calling the tools + piping the results through Semantic), and use it in 
the Semantic xref backend. So the results will be displayed by xref.

Alas, xref is not up to feature parity yet (cannot group hits by 
containing functions, no checkboxes, simplistic "rename" command), and 
I'm having difficulties deciding on the changes to the xref data 
structures that would make that possible, as well as keep allowing 
certain other uses of xref, such as project-find-regexp.

I'll write that up in a separate thread later this week.

> But as
> long as it's there, I'd appreciate if you could leave whatever support
> you have coded for it.

Just to be clear, there's no "support" for semantic-symref in xref yet. 
We just use some low-level pieces of it as a "fast Grep".

> I'd also be happy to drop the completion GUI stuff, BTW. I've repeatedly
> said that company should be in core for that.

Also not exactly necessary. Semantic should make its 
completion-at-point-functions element the main entry point for 
completion, and Company will pick it up. You can even have argument list 
annotations, if you add the :annotation-function property.

I do want to have Company more widely available, but probably not in 
25.1. There are quite a few things I wanted to do with it first, and 
most are still untouched.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:34 PM, Eli Zaretskii wrote:

> Grep doesn't scale well to large projects, IME.  You get too many
> false positives.

We can also check the "inside string or comment" status, as well as use 
the Emacs syntax tables to make sure that the match begins and ends at 
symbol boundaries appropriate for the file's language. Everything 
necessary for that is already written (and you can try it out by calling 
project-find-regexp and using something like \_<tool_bar_items\_> as the 
regexp). I don't think that id-utils does anything more to weed out 
false positives.

Grep is still probably going to be slower than at least some of the 
tools in question. Could you test, on a large project of your choice?

> Outdated databases are easy to avoid with the likes of cron jobs.
> Yes, that's hand-holding, but when you have to quickly find stuff in a
> project with 3 million lines of code and thousands of classes, there
> really is no other alternative.

Yes. But the xref commands should be easy to use. Even if the above is 
not rocket science, the user would still have to know what they need to 
do to get up-to-date results. (Believe it or not, I haven't created a 
single cron job for code writing purposes in my life, and I don't know 
its syntax for the time intervals. I'm likely not alone in that.)

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 11:16 PM, Dmitry Gutov wrote:

> project-find-regexp and using something like \_<tool_bar_items\_> as the

Sorry, using the Emacs-only syntax there doesn't exactly work yet. But 
you can compare by calling xref-find-references when the project has an 
id-utils index file, and doing the same without it.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:35 PM, John Wiegley wrote:
>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Great, let's do that. I'd prefer to have the functionality of xref.el in core.

Let me know when it becomes a blocker, and I'll do it in the simplest 
way. But I'd like to defer that for now, because it comes with making 
choices (see the other messages in this thread), and I'm running low on 
decision-making juice these days.

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> ELPA packages that other core code depends on (like CEDET; xref uses it -
>> called "core ELPA packages" hereinafter) have to be in every developer's
>> build environment everyday; the other core code has to see the current
>> version of the package, and it has to be the same for every developer.
>> 
>> If core ELPA packages are in the ELPA git repository, you would copy some
>> subtrees of the ELPA git workspace into the Emacs git workspace. People have
>> looked into doing this, but it gets messy and confusing. "git fetch" is
>> dealing with two upstreams for one workspace. There has been talk about git
>> submodules, etc, but nothing concrete.
>
> Elpa.git should be a submodule referenced from within Emacs.git (under
> "elpa").

That's overkill; there are many packages in Gnu ELPA that core code
should _not_ depend on.

We could try to have a list of approved packages in a file somewhere,
but I don't think we can enforce that; better to use git to enforce it.

We need a way to explicitly label and enforce which ELPA packages core
code can depend on. The only demonstrated way to do that is to keep them
in Emacs git.

> This allows us to expressly state which "version" of Elpa is expected
> to work with the current Emacs.git.

That's only required for the ELPA packages that core code can depend on.
For the other ELPA packages, we explicitly don't want to know if they
work with Emacs master; that's up to the individual package maintainers.
That's one of the primary purposes of ELPA.

> Large packages like CEDET should move outside of Emacs.git and into
> Elpa.git.

If this said "large packages that the rest of Emacs core does not depend
on ...", I could agree.

I just checked; the only other core code that depends on CEDET is
xref.el, which can be fixed. So I agree, CEDET should move to elpa git.

However, I assume the reason CEDET is in Emacs git is because there is a
desire to distribute it in the tarball; a sufficient number of users
expect it to be there without going thru package install. So the
mechanism to put ELPA packages in the tarball (but _not_ in the emacs
development workspace) must also be implemented. That should not be too
hard; no git submodules, just directory tree copy.

So this introduces a third kind of ELPA package: "distributed in Emacs
tarball". A "tarball package" for short?

Except that Eric maintains that there is more to "CEDET" in Emacs core
than just lisp/cedet/*. So to be clear, we are proposing to move only
lisp/cedet/* to elpa git.

> If xref.el depends on CEDET, it would move to Elpa.git as well.

As Dmitry points out, better to remove the dependency from the core
xref.el. Then we can add it back either in the ELPA CEDET package or in
a new xref-cedet ELPA package.

>> If we use the approach of keeping core ELPA packages in the Emacs git
>> repository, there is _zero_ friction for the current core Emacs developers;
>> the only change is in the ELPA config scripts.
>
> I do not want this code replication. 

How is it replication?

There are two styles of ELPA packages currently:

1) The only repository for the code is Gnu ELPA git

2) There is an external repository that the package maintainer uses;
   releases are pushed to Gnu ELPA git

The choice is up to the ELPA package maintainer.

The proposal is to change both to say:

1a) The only repository for the code is Gnu ELPA git or Gnu Emacs git.

2a) There is an external repository that the package maintainer uses;
    releases are pushed to Gnu ELPA git or Gnu Emacs git.

The ELPA packages in Emacs git are "core ELPA packages"; core Emacs code
may rely on them. 

There is no additional replication by keeping the core ELPA packages in
Emacs git.

The only reason the core Emacs packages are in the ELPA server is to
allow package releases in addition to Emacs releases. The package
maintainers have to accept maintaining backward compatibility, so a core
package in an Emacs 25.1 binary install can work with a more recent
version of the core ELPA package.

In summary, I'm proposing that there are three kinds of ELPA packages:

1) "normal ELPA package". 

    - Stored in Gnu ELPA git, and possibly in the package maintainer's
      separate repository.

    - Releases only occur via the ELPA package server, whenever the
      version number increments in the ELPA git main package file.
    
    - Core Emacs code may not depend on these.

2) "tarball ELPA package". 

    - Stored in Gnu ELPA git, and possibly in the package maintainer's
      separate repository.

    - Each Emacs release includes a released version of the package in
      the Emacs tarball, via copy from an ELPA workspace at Emacs
      tarball build time. Other releases also occur, via the ELPA
      package server, whenever the version number increments in the ELPA
      git main package file.
    
    - Core Emacs code may not depend on these.

3) "core ELPA package".  

    - Stored in Gnu Emacs git, and possibly in the package maintainer's
      separate repository.

    - Each Emacs release includes a version of the package in the
      tarball, because it is in the Emacs git workspace. Other releases
      also occur, via the ELPA package server, whenever the version
      number increments in the Emacs git main package file.
    
    - Core Emacs code may depend on these.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> However, like mentioned in the "project.el semantics" thread:

please change the thread subject.

-- 
-- Stephe

Re: xref CEDET (was ELPA policy)

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Date: Tue, 10 Nov 2015 10:35:13 -0800
>> Cc: aaronecay@gmail.com, Eli Zaretskii <eliz@gnu.org>, Stromeko@nexgo.de,
>> 	Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
>> 
>> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>> > I can cut out xref.el's dependency on CEDET easily enough.
>> 
>> Great, let's do that.
>
> Sounds like a step backward to me.  CEDET allows xref to be more
> versatile and support more back-ends.

I will provide an ELPA package that implements a CEDET backend for xref.

Or, if it's simple enough, add that to the CEDET ELPA package.

This issue is not a show-stopper to moving CEDET to ELPA.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>
>> Why?
>
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA. 

I'm happy to have someone making decisions, but there needs to be some
rationale given, so we understand the tradeoffs.

I believe the rationale here is:

- CEDET is not used by other core Emacs code (xref.el will be fixed)

- code that is not used by other core Emacs code clutters the workspace,
  producing extra hits in greps, etc.

- CEDET can still be included in the Emacs tarball, so users do
  not have to use list-packages to install it (this requires a to be
  implemented mechanism).

>> From my experience, git submodules are never seamless to work with.
>
> Even still, they work perfectly fine for this purpose.

I would like to see an actual demonstration, please; there are lots of
devils in the details.

Which means:

Please post a clear recipe for setting up an Emacs + ELPA developer
workspace, starting from git clone, so I can try this on my machine.


The test is then that some core Emacs code depend on some ELPA package;
we'd need branches in both to make that happen, since it doesn't now.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>> This is not about reaching a consensus. This is about you giving proper
>> reasons for such a big change.
>
> To be clear, I would also put Eshell (though not pcomplete) into the category
> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
> be in the release tarball.
>
> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
> in ELPA today, and we were talking about moving it into core, I would have
> just as much trouble justifying that move too. Perhaps this simply underscores
> the fact that we don't have an agreed upon ELPA policy we can all
> refer to.

Yes.

> OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
> will be, until it becomes clear, based on that document, what should go where,
> and why. We'll defer discussions of package movement until we have that in
> place.

Thank you.


-- 
-- Stephe

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

>> Elpa.git should be a submodule referenced from within Emacs.git (under
>> "elpa").

> That's overkill; there are many packages in Gnu ELPA that core code should
> _not_ depend on.

I believe that no core code should depend on any ELPA packages. Such a
dependency would be a reason to move that package into core, no? If that's
really the case, no submodule reference is necessary after all.

> If this said "large packages that the rest of Emacs core does not depend on
> ...", I could agree.

Yes, I meant that.

> So this introduces a third kind of ELPA package: "distributed in Emacs
> tarball". A "tarball package" for short?

Yes, for ELPA packages, there would be two flavors: in the tarball, and not.
Some have suggested just putting everything in the tarball for now. So we
could defer this question until the size of the tarball mattered, or if a
particular package was so dynamic it didn't want a version frozen into the
tarball.

Note that going down this road starts to make the line between "core" and
"ELPA" very thin, with easy migration between the two when desired. For
developers it allows Git to focus better, but for users, the difference is
largely invisible.

> Except that Eric maintains that there is more to "CEDET" in Emacs core than
> just lisp/cedet/*. So to be clear, we are proposing to move only
> lisp/cedet/* to elpa git.

Yes.

> In summary, I'm proposing that there are three kinds of ELPA packages:

I think we have a lot of agreement here, but I'd still like a clearer policy
than just "if core depends on it".

John

Re: ELPA policy

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> John Wiegley writes:
>>>>>>> David Engster <deng@randomsample.de> writes:
>>
>>> This is not about reaching a consensus. This is about you giving proper
>>> reasons for such a big change.
>>
>> To be clear, I would also put Eshell (though not pcomplete) into the category
>> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
>> be in the release tarball.
>>
>> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
>> in ELPA today, and we were talking about moving it into core, I would have
>> just as much trouble justifying that move too. Perhaps this simply underscores
>> the fact that we don't have an agreed upon ELPA policy we can all refer to.
>
> In my opinion, the main question is whether something provides
> infrastructure for other packages to use. 

I think it is a reasonable goal to allow ELPA packages to serve in that
role, as long as the "other packages" are also normal or tarball ELPA
packages.

> This is precisely what CEDET tries to do.

Yes.

However, now the shoe is on the other foot: why must infrastructure
packages be in Emacs core?

I think the trigger is "some other _core_ code wants to depend on it".
That would trigger a move to Emacs git, as a core ELPA package (the
package could still have intermediate released via the Gnu ELPA server).

> I wouldn't have much trouble with putting parts of CEDET in ELPA,
> namely those parts that do not directly provide infrastructure, like
> support for certain languages, project types, indexing tools, etc.

Good, but let's not try to do that for Emacs 25; since we are trying to
get to feature freeze, it's too much.

> It is still not clear to me what exactly is gained by moving core
> packages to ELPA. 

One gain is making it clear that other core code is not allowed to depend
on it. This is in turn to ensure that it doesn't creep into the dumped
code. But I'm not sure that's an important reason.

-- 
-- Stephe

RE: ELPA policy

[Drew Adams]

Here's a naive question regarding where you place Lisp code
that is distributed with Emacs (part of the tarball): Will
anything change for users, depending on where you happen to
manage the code wrt ELPA etc.?

I'm guessing (and hoping) not, so that the distributed
directory structure will remain the same.  Is that correct?

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>> the main question is whether something provides infrastructure for other
>> packages to use.
>
> Sounds like a good sentence for an ELPA policy. :)
>
>> It is still not clear to me what exactly is gained by moving core packages
>> to ELPA.

Since no other core code depends on ELPA, CEDET is _not_ a "core package".

Rather, CEDET is a "tarball package"; it is in Emacs git solely to
ensure that it is included in the Emacs release tarball.

A better example of a possible "core ELPA package" is the "seq" package.

> Agility. 

I hear that as "easier to make small/frequent changes". That is what the
ELPA release policy gives you, yes. So this is one rationale for moving
packages to ELPA.

> What is appropriate. 

That's what we are trying to figure out :)

> Knowing when a thing goes into core, and when in ELPA.

Ditto.

> Org is an application, it's not infrastructure; the same with Gnus. *Parts* of
> Gnus might rightly be considered infrastructure, but the whole of Gnus just
> doesn't belong there. Parts of CEDET probably do belong in Emacs core, but as
> an application, I don't think the whole of it does.

I think the distinction between "tarball package" and "core package" is
helpful here.

I'm guessing that the main motivation for including Org and Gnus in
Emacs git (well, CVS back then?) was to include them in the release
tarball. If we have a mechanism to allow that for ELPA packages, moving
them to ELPA makes sense.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>> aaronecay@gmail.com, Eli Zaretskii <eliz@gnu.org>,
>> Stromeko@nexgo.de, Dmitry Gutov <dgutov@yandex.ru>,
>> emacs-devel@gnu.org
>> Date: Tue, 10 Nov 2015 10:52:41 -0800
>> 
>> > Why?
>> 
>> There will never be 100% agreement on whether they should be in ELPA, or be in
>> Core, so I'm making the decision that they belong in ELPA.
>
> IMO, it's a mistake to move CEDET.

Bald statements of preference, without rationale, do not help move this
discussion forward.

We are trying to establish a general ELPA policy, not just vote on one
particular package.

What is it about CEDET that makes it a poor candidate?

How would your workflow suffer if it was moved?

How would users suffer?

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, aaronecay@gmail.com,
>> Stromeko@nexgo.de, emacs-devel@gnu.org, Dmitry Gutov
>> <dgutov@yandex.ru>
>> Date: Tue, 10 Nov 2015 10:28:26 -0800
>> 
>> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.
>
> "Should" based on what? just the fact that it's large?  I think we
> should decide this stuff on a case by case basis.  For example:
>
>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
> IMO, the exact opposite: if there are core features that we want to be
> in Emacs no matter what, and those features depend on a package which
> could be a candidate to move to ELPA, that package should NOT move to
> ELPA.

In general, I agree.

However, there are two _independent_ aspects of "move to ELPA":

1) Move to ELPA git.

2) Be distributed by the ELPA package server.


If other core code depends on a package, it should not move to ELPA git
(unless the dependency is easily refactored, as in the CEDET/xgit case;
there will always be corner cases).

However, if there is a desire to do package releases between Emacs
releases, making the package available via the ELPA package server makes
sense.

I think this discussion is mainly focused on "move to ELPA git", but it
would help to be clear.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>> "Should" based on what? just the fact that it's large? I think we should
>> decide this stuff on a case by case basis. For example:
>
> I'm surprised you say this, Eli, because in another thread you agree that
> packages like this should be in Elpa, didn't you?
>
>>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
>> IMO, the exact opposite: if there are core features that we want to be in
>> Emacs no matter what, and those features depend on a package which could be
>> a candidate to move to ELPA, that package should NOT move to ELPA.
>
> Core should provide functionality along the lines of a "standard library" and
> a "standard environment", where having them in core is as much a statement
> about consistency of interface, as it is about universal availability of the
> functionality.

Right. Since CEDET provides lots of interesting infrastrucure, I assumed
it was part of the Emacs standard library.

But I also see no reason not to have two levels of "standard
infrastructure"; one in core Emacs, on in ELPA.

That said, moving CEDET out of core to ELPA can be seen as a demotion.
For example, I was assuming that the EDE package system should be
prefered over any package system that is in ELPA, precisely because it
is in Emacs git. So I was spending my time improving EDE, rather than
some other ELPA package.

That aspect should be considered.

In other emails, I've said "CEDET is a tarball package; in
Emacs git solely to be included in the tarball".

Is it fair to add "also because it is approved as part of the standard
Emacs library"?

> Since xref.el does not need to depend on CEDET, I don't see a reason why it
> should, causing CEDET to remain in core.

I would object to moving CEDET to ELPA git if it were not easy to
refactor the current dependency to still be provided by an ELPA package.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Since xref.el does not need to depend on CEDET, I don't see a reason why it
>> should, causing CEDET to remain in core.
>
> xref is as useful as its backends.  If you take away backends, it
> becomes less useful, or supports less programming languages, or both.

Yes.

That's an argument for making the CEDET xref backend available
somewhere; I've committed to ensuring that.

But this discussion is about ELPA vs Emacs core:

Does it matter whether that backend is in ELPA or Emacs core?

-- 
-- Stephe

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> I think the distinction between "tarball package" and "core package" is
> helpful here.
> 
> I'm guessing that the main motivation for including Org and Gnus in Emacs
> git (well, CVS back then?) was to include them in the release tarball. If we
> have a mechanism to allow that for ELPA packages, moving them to ELPA makes
> sense.

I like this. I think we have a good striation:

  core
  tarball ELPA
  net ELPA

To the user, core and tarball ELPA should be indistinguishable. I think also
that some tarball ELPA packages should come "pre-installed", if that is not
already done. This would makes them accessible via autoload, rather than
requiring the package interface to opt-in.

If core == tarball ELPA for everyone but us, this makes the decision easy
whether something should be in core or not a lot: We put it in core if core
requires it. If nothing at all in core uses the package -- and if that package
doesn't define "essential" functionality, like isearch.el or grep.el -- it can
shift to tarball ELPA.

This still leaves open the meaning of "essential", but it does make the
picture clearer.

John

RE: ELPA policy

[Drew Adams]

> > I think the distinction between "tarball package" and "core package" is
> > helpful here.
> >
> > I'm guessing that the main motivation for including Org and Gnus in
> > Emacs git (well, CVS back then?) was to include them in the release tarball.
> > If we have a mechanism to allow that for ELPA packages, moving them to ELPA
> > makes sense.
> 
> I like this. I think we have a good striation:
>   core
>   tarball ELPA
>   net ELPA
> To the user, core and tarball ELPA should be indistinguishable.

I haven't yet received an answer to my question whether anything
will change for users, depending on where you happen to manage
the code wrt ELPA etc.

But it sounds like the answer is yes.  If some stuff that has
traditionally been part of the distribution gets moved to (net)
ELPA, it will no longer be distributed to users.  They will
need to pull it down using the package interface.  Is that right?

If so, I'm not crazy about that.  I don't particularly want to
go fishing in (net) ELPA for stuff that I've always been able to
simply grep from within the distribution `lisp' directory. 
Especially, but not only, when I am not on the Internet.

I hope you will continue to (also) distribute Emacs with all
of its (traditional) source code, and not just make users
request it from (net) ELPA.

Re: ELPA policy

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> I haven't yet received an answer to my question whether anything will change
> for users, depending on where you happen to manage the code wrt ELPA etc.
> 
> But it sounds like the answer is yes. If some stuff that has traditionally
> been part of the distribution gets moved to (net) ELPA, it will no longer be
> distributed to users. They will need to pull it down using the package
> interface. Is that right?

Correct. Sorry I assumed that the discussion had answered your question in
passing.

> If so, I'm not crazy about that. I don't particularly want to go fishing in
> (net) ELPA for stuff that I've always been able to simply grep from within
> the distribution `lisp' directory. Especially, but not only, when I am not
> on the Internet.

Those things could then be in tarball ELPA. The distinction between tarball
ELPA and net ELPA is much more plastic than core and tarball ELPA. In fact, I
have a feeling that, initially, tarball ELPA and net ELPA will just be the
same thing, until we have a reason to tease them apart.

> I hope you will continue to (also) distribute Emacs with all of its
> (traditional) source code, and not just make users request it from (net)
> ELPA.

This is my preference as well.

John

Re: ELPA policy

[Stephen Leake]

Drew Adams <drew.adams@oracle.com> writes:

> Here's a naive question regarding where you place Lisp code
> that is distributed with Emacs (part of the tarball): Will
> anything change for users, depending on where you happen to
> manage the code wrt ELPA etc.?
>
> I'm guessing (and hoping) not, so that the distributed
> directory structure will remain the same.  Is that correct?

That's a good point. But I'm not clear what previous state you are
comparing to.

There are two cases;

1) A package that is currently in Emacs git moves to Gnu ELPA git but
   remains in the Emacs tarball.

2) A package that is currently in Gnu ELPA git stays there, and is added
   to the Emacs tarball.

I would argue that tarball ELPA packages should be installed in some
system level elpa directory (similar to the user's .emacs.d/elpa), not
in the main emacs/lisp install directory.

I don't think the history of a package should determine where it is
installed. So in case 1), I expect the package to be in a different
installation directory from the previous Emacs release. But it's still
in the default load-path, so this change is transparent to the user.

But it's not too important; the important distinction is that tarball
ELPA packages are _not_ in the developer Emacs workspace, so core
code cannot accidently depend on them.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

> The distinction between tarball
> ELPA and net ELPA is much more plastic than core and tarball ELPA. In fact, I
> have a feeling that, initially, tarball ELPA and net ELPA will just be the
> same thing, until we have a reason to tease them apart.

Including a package in tarball ELPA means that the Emacs project has
decided that the package is sufficiently important to be made
immediately available to all Emacs users with no effort on their part
beyond installing Emacs from the tarball.

That implies a certain level of maturity and quality, and implies that
it should be favored over other similar packages.

On the other hand, we have in several cases recommended that someone put
a package in Gnu ELPA in order to give it more exposure, so that it
might eventually become higher quality and more mature.

In general, ELPA packages have far less oversight than core packages.
tarball ELPA packages should have the same oversight as core, or at
least more than normal ELPA packages.

So I disagree that all of Gnu ELPA should be included in the tarball.

Another choice would be to say "use MELPA for experimental packages",
but it is already too late for that.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

I do not consider myself competent enough in this area, but I'd like
to share some thoughts here:

For each package version there is a range (possibly sparse) of core
versions on which the package is supported (or just should work).

While the intent to move as much as possible in ELPA can be understood,
it leads to potentially more incompatibilities between important parts
which can provide API by themselves.

So, there should be some compromise between "latency" of new features
before they generally become available for use in packages and overall
core stability.  To me, it seems that the latter is more important and
it's better to keep infrastructure & library things in core, while
moving everything that uses them for a concrete purpose to ELPA.

If that in turn provides some API which others use, then we have
package interdependencies and that is probably OK (but can lead to
conflicts).  If sufficient amount of important packages use some API
and that API is rather mature, then the maintainer can decide to move
that in core to simplify dependency management.

So, I'd suggest that:

- Language features go straight into core (and people working on them
  / using them will have to use git version of Emacs until next
  release)

- New libraries which are not supposed to be in very common use go to
  ELPA first

- Then they are probably promoted to core as they get mature and more
  widely used - just to simplify their usage ("they will always be
  available").

- Major applications (like Gnus) and smaller ones always go to ELPA
  (most probably we should also bundle them in a tarball, but keep
  outside of the core).  A user can then decide whether to use git
  version of e.g. Gnus (from Elpa or private package repository) if she
  likes, or update to a released package version from Elpa (if core
  supports it), or just keep using the Elpa version she uses at the
  moment (which probably came together with current core).

- If such an application provides things useful for other
  applications, then it probably should extract that into a library to
  go through the cycle oulined above (elpa --maturity--> core).

The API / SPI notion can also be used to provide implementations
(backends) for different features which may have default simple
implementations in core and more advanced ones in packages.  A package
must somehow "register" itself as a candidate for being a service for
a concrete feature during installation.

Filipp

RE: ELPA policy

[Drew Adams]

> > Here's a naive question regarding where you place Lisp code
> > that is distributed with Emacs (part of the tarball): Will
> > anything change for users, depending on where you happen to
> > manage the code wrt ELPA etc.?
> >
> > I'm guessing (and hoping) not, so that the distributed
> > directory structure will remain the same.  Is that correct?
> 
> That's a good point. But I'm not clear what previous state you are
> comparing to.

Previous delivery of Emacs (back to ~Day One), including its
source code, especially Lisp, under directory lisp/.  What's
unclear about that?

> There are two cases;
> 1) A package that is currently in Emacs git moves to Gnu ELPA git but
>    remains in the Emacs tarball.
> 2) A package that is currently in Gnu ELPA git stays there, and is added
>    to the Emacs tarball.
> 
> I would argue that tarball ELPA packages should be installed in some
> system level elpa directory (similar to the user's .emacs.d/elpa), not
> in the main emacs/lisp install directory.

You say you would argue that, but where's the argument?  How about
a reason?

As one user, I would prefer that all distributed Lisp code be under
lisp/.  Makes things simpler for users (who might even have existing
scripts etc. that expect this).

Is there a good reason not to keep this behavior?

I don't really care where you stick code, or how you access it, for
development purposes.  Where an Emacs distribution puts it for
_users_ should remain what it has been, IMO, if possible - somewhere
under lisp/, at least.

Why should changes in where you keep code for development affect
where users of Emacs find it?

> I don't think the history of a package should determine where it is
> installed. So in case 1), I expect the package to be in a different
> installation directory from the previous Emacs release. But it's still
> in the default load-path, so this change is transparent to the user.

`load-path' is not everything, so no, it is _not_ transparent to
users, depending on what they do and how they use Emacs (including
its source code).

Emacs should be able to present its source code to users in the
same way as in the past, I would think.  They certainly should
not be _required_ to use the package system or git or ... to
access it.

`package.el' should be a nice-to-have.  It should not become an
obligatory hoop for users to jump through.  Most users will find
it handy, just as some users might find a menu handy.  But it
should not become a necessary way to access Emacs code.

> But it's not too important; the important distinction is that tarball
> ELPA packages are _not_ in the developer Emacs workspace, so core
> code cannot accidently depend on them.

That might be important to developers, but it is not the issue I
asked about.

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Richard Stallman <rms@gnu.org> writes:

> It is fine to have it there, but this suggests we should maintain
> it inside Emacs.

I'm not sure I understand what you mean Richard.  seq.el is maintained
inside Emacs (in lisp/emacs-lisp/seq.el and
test/automated/seq-tests.el).  The version published in ELPA is there
for backward-compatibility, and does not have all the features (some
features do require Emacs 25).

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

xref and GNU Global (Re: ELPA policy)

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:


> It doesn't support CEDET even now. Nobody wrote the Semantic 
> implementation of a xref backend, and it might be tool late now for the 
> 25.1 release.

What about having support for Global (I haven't followed the development
of xref in a while, so I might have missed it).  I'm using ggtags now,
but having support for it in xref would be awesome.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: xref and GNU Global (Re: ELPA policy)

[Dmitry Gutov]

On 11/11/2015 05:15 PM, Nicolas Petton wrote:

> What about having support for Global (I haven't followed the development
> of xref in a while, so I might have missed it).  I'm using ggtags now,
> but having support for it in xref would be awesome.

It's the other way around: ggtags will need to implement support for xref.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Another choice would be to say "use MELPA for experimental packages",
  > but it is already too late for that.

If my memory serves, MELPA contains packages we would like to include
in Emacs, but we can't get legal papers for them.
We are seeking people to write replacements for those, though we wish
it were not necessary.

We would like developers of future useful packages to want to put
their packages into the GNU ELPA.

To recommend putting some packages -- whatever they might be -- into
MELPA would work against these goals.  It would have been a bad
approach.

  > Which raises the question; what is the rationale for moving from MELPA
  > to Gnu ELPA?

If a package is useful to recommend, and we CAN get papers for it,
we definitely want to move it to GNU ELPA.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It is fine to have it there, but this suggests we should maintain
  > > it inside Emacs.

  > I'm not sure I understand what you mean Richard.

I mean that we should maintain the file inside Emacs, 
not inside ELPA.

  >   seq.el is maintained
  > inside Emacs (in lisp/emacs-lisp/seq.el and
  > test/automated/seq-tests.el).

That means people are already doing what I think we should do.
I didn't have this information when I wrote my previous message.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> If a package is useful to recommend, and we CAN get papers for it, we
> definitely want to move it to GNU ELPA.

Yes, this is why I want to clearly define ELPA policy, and streamline things
as much as possible for developers and users: so that it becomes a more
attractive means for distributing Emacs packages. I think many people may be
largely ignoring it right now, and so they reach for MELPA. Since so many
people contribute to MELPA, they consider it a more attractive distribution
platform.

John

RE: ELPA policy

[Drew Adams]

> We would like developers of future useful packages to want to put
> their packages into the GNU ELPA.
> 
> To recommend putting some packages -- whatever they might be -- into
> MELPA would work against these goals.  It would have been a bad
> approach.

FWIW -

I will continue to use MELPA for my libraries, and not GNU ELPA,
for one reason:

I post my code to Emacs Wiki (to locked pages).  MELPA pulls
the code from the wiki (daily).  I do not need to use GIT etc.

This is the way I prefer to work.  Call it old-fashioned or
bizarre; I don't care.

So you might recommend that packages not be put in MELPA, but
some will continue to be put there, including perhaps some that
you might someday want to include in Emacs.

>   > Which raises the question; what is the rationale for moving from
>   > MELPA to Gnu ELPA?
> 
> If a package is useful to recommend, and we CAN get papers for it,
> we definitely want to move it to GNU ELPA.

This was refused by Stefan for my libraries (for which FSF has papers),
based on the fact that I do not use GIT, so any updates I make to them
would not be done directly in the repository.  It was not acceptable to
update elsewhere (e.g. the wiki) and then have someone or a program
pull from there to the repository when appropriate.  So be it.

Re: ELPA policy

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> I will continue to use MELPA for my libraries, and not GNU ELPA, for one
> reason:
>
> I post my code to Emacs Wiki (to locked pages). MELPA pulls the code from
> the wiki (daily). I do not need to use GIT etc.
>
> This is the way I prefer to work. Call it old-fashioned or bizarre; I don't
> care.

This is a legitimate complaint, Drew. One of the things I want to avoid with
ELPA is making it harder for people to contribute to than other alternatives.

Since pulling in data from the Web indiscriminately is not possible for this
project (we have to make sure we're not unwittingly including code without a
proper copyright assignment), perhaps we need as an "integrator": someone
willing to guide the update of ELPA from various sources on the Web, keeping
an eye out for changes that might affect copyright.

It wouldn't be too hard to setup a process that updates a local clone of ELPA
with all your latest versions from GitHub and EmacsWiki, and then to check
over a diff of those changes before pushing them to ELPA. One might even
suggest that you could do this yourself, as part of your own release work.
Would that be something you'd find worth doing?

John

RE: ELPA policy

[Drew Adams]

> This is a legitimate complaint, Drew. One of the things I want to avoid
> with ELPA is making it harder for people to contribute to than other
> alternatives.
> 
> Since pulling in data from the Web indiscriminately is not possible for
> this project (we have to make sure we're not unwittingly including code without
> a proper copyright assignment), perhaps we need as an "integrator": someone
> willing to guide the update of ELPA from various sources on the Web,
> keeping an eye out for changes that might affect copyright.

I'm glad to hear that; all of it.  And thanks for replying.

> It wouldn't be too hard to setup a process that updates a local clone of
> ELPA with all your latest versions from GitHub and EmacsWiki, and then to check
> over a diff of those changes before pushing them to ELPA. One might even
> suggest that you could do this yourself, as part of your own release work.
> Would that be something you'd find worth doing?

I don't want to develop anything like that; sorry.  I don't want to fiddle
with a local GIT clone.  I would be willing to identify versions of my
libraries to be pulled at specific times ("releases") from the wiki (or
from MELPA) to be put (updated) in GNU ELPA.

I doubt that my case is typical, however.  I'm not saying that Emacs Dev
needs to cater to such a case.  I'm just explaining that I update my
libraries locally, without GIT etc., and I upload updates to the wiki.
They are automatically pulled to MELPA.

What do I need to know, currently?  Only how to write some Lisp (a
library) and how to copy and paste it into a Web page.  Simple.  Maybe
too simple to accommodate by Emacs Dev.

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> For each package version there is a range (possibly sparse) of core
> versions on which the package is supported (or just should work).

For normal ELPA packages, that's true.

For core and tarball ELPA packages, they must work in each current Emacs
release (since they are in the release tarball). They may also support
some set of previous Emacs releases.

> While the intent to move as much as possible in ELPA can be understood,
> it leads to potentially more incompatibilities between important parts
> which can provide API by themselves.

Yes.

> So, there should be some compromise between "latency" of new features
> before they generally become available for use in packages and overall
> core stability.  To me, it seems that the latter is more important and
> it's better to keep infrastructure & library things in core, while
> moving everything that uses them for a concrete purpose to ELPA.

For normal ELPA packages, that's true. But this is partly why core and
tarball ELPA packages are being considered.

> If that in turn provides some API which others use, then we have
> package interdependencies and that is probably OK (but can lead to
> conflicts). 

A more flexible requirements spec may  be needed. For example, Debian
packages allow specifying a range of versions in the dependency; that
can reflect actual testing, and not rely on implicit promises of future
backward compatibility. Of course, it can also lead to false failures.

> If sufficient amount of important packages use some API and that API
> is rather mature, then the maintainer can decide to move that in core
> to simplify dependency management.

or to a tarball ELPA package, or to a core ELPA package.

Perhaps that is too much choice?

> So, I'd suggest that:
>
> - Language features go straight into core (and people working on them
>   / using them will have to use git version of Emacs until next
>   release)

By "language features" do you mean things like ada-mode.el (primary mode
for editing Ada source files)? If so, I strongly disagree; ada-mode.el
is very happy as a normal ELPA package. Similar modes for other
languages with a small audience could be in ELPA as well; I have not
looked to see what's actually there.

cc-mode is the core for a lot of similar C-like languages, so it
probably makes sense to keep that in core. But it would be interesting
to see what the consequences of moving it to a normal or tarball ELPA
package would be.

But if you mean things like parsers and xref support, then I agree.
Although ada-mode also provides its own parser ...

> - New libraries which are not supposed to be in very common use go to
>   ELPA first

> - Then they are probably promoted to core as they get mature and more
>   widely used - just to simplify their usage ("they will always be
>   available").

That's what tarball ELPA packages are for.

The only reason to move a package to actual core (ie emacs git, as
opposed to elpa git) is if some other core code wants to depend in it.

> - Major applications (like Gnus) and smaller ones always go to ELPA
>   (most probably we should also bundle them in a tarball, but keep
>   outside of the core).

Right; that's a tarball ELPA package.

> - If such an application provides things useful for other
>   applications, then it probably should extract that into a library to
>   go through the cycle oulined above (elpa --maturity--> core).

Yes.

> The API / SPI notion can also be used to provide implementations
> (backends) for different features which may have default simple
> implementations in core and more advanced ones in packages.  A package
> must somehow "register" itself as a candidate for being a service for
> a concrete feature during installation.

If we use cl-generic to provide dispatching, the cl-defmethod form does
the registration.

-- 
-- Stephe

Re: xref and GNU Global (Re: ELPA policy)

[Stephen Leake]

Nicolas Petton <nicolas@petton.fr> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>
>> It doesn't support CEDET even now. Nobody wrote the Semantic 
>> implementation of a xref backend, and it might be tool late now for the 
>> 25.1 release.
>
> What about having support for Global (I haven't followed the development
> of xref in a while, so I might have missed it).  I'm using ggtags now,
> but having support for it in xref would be awesome.

I'm working on it, but I seem to be going very slowly.

-- 
-- Stephe

Re: xref and GNU Global (Re: ELPA policy)

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/11/2015 05:15 PM, Nicolas Petton wrote:
>
>> What about having support for Global (I haven't followed the development
>> of xref in a while, so I might have missed it).  I'm using ggtags now,
>> but having support for it in xref would be awesome.
>
> It's the other way around: ggtags will need to implement support for xref.

ggtags is a fairly thin elisp package that interfaces to command-line
Gnu Global.

cedet also provides a thin elisp package (cedet-global) that interfaces to
command-line Gnu Global, and is integrated with the rest of cedet.

There's no reason to have both cedet-global and ggtags.

There are many reasons to keep cedet, so it makes sense to provide xref
support for Gnu Global via the cedet package, not the ggtags package.

There may be features that are in ggtags that are not in cedet; I'll
have to check after I get the basics working, and see if it is
reasonable to add them to cedet.

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since pulling in data from the Web indiscriminately is not
  > possible for this project (we have to make sure we're not
  > unwittingly including code without a proper copyright assignment),
  > perhaps we need as an "integrator": someone willing to guide the
  > update of ELPA from various sources on the Web, keeping an eye out
  > for changes that might affect copyright.

What we need to solve Drew's problem is to merge changes into a few
specific files from a specific place, and commit them using git.
Isn't that easy enough to do?

There could be one page where he provides the commit log info.
Whenever that page changes (we could check every 5 minutes), our demon
could merge in all the files which have changed.  They it would email
the diffs to Drew so he could confirm what he installed.

Thus, when Drew wants to install a new version, he would change the
other pages first, then write the change log text into the log page.
Then wait 5 minutes and it's done.

This is assuming there are no other obstacles aside from the
mechanics of installing changes into our repository.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: xref and GNU Global (Re: ELPA policy)

[Dmitry Gutov]

On 11/12/2015 01:19 AM, Stephen Leake wrote:

> There's no reason to have both cedet-global and ggtags.

ggtags has some other things as well, like a 
completion-at-point-functions element, hooks to update the current 
database automatically, eldoc support, etc.

> There are many reasons to keep cedet, so it makes sense to provide xref
> support for Gnu Global via the cedet package, not the ggtags package.

We can have either, as far as I'm concerned. But it would be better to 
have more features from ggtags in the core to replace the things 
etags.el is commonly used for, eventually.

> There may be features that are in ggtags that are not in cedet; I'll
> have to check after I get the basics working, and see if it is
> reasonable to add them to cedet.

CEDET only uses Global is a fairly narrow way.

Re: xref and GNU Global (Re: ELPA policy)

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

>> What about having support for Global (I haven't followed the development
>> of xref in a while, so I might have missed it).  I'm using ggtags now,
>> but having support for it in xref would be awesome.
>
> I'm working on it, but I seem to be going very slowly.

Do you want help?

BTW, regarding CEDET, I never tried it, but I don't plan to use it
either.  I think the best would be to just add a simple backend for
xref, with no dependency to CEDET.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

RE: ELPA policy

[Drew Adams]

>   > Since pulling in data from the Web indiscriminately is not
>   > possible for this project (we have to make sure we're not
>   > unwittingly including code without a proper copyright assignment),
>   > perhaps we need as an "integrator": someone willing to guide the
>   > update of ELPA from various sources on the Web, keeping an eye out
>   > for changes that might affect copyright.
> 
> What we need to solve Drew's problem is to merge changes into a few
> specific files from a specific place, and commit them using git.
> Isn't that easy enough to do?
> 
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.
> 
> Thus, when Drew wants to install a new version, he would change the
> other pages first, then write the change log text into the log page.
> Then wait 5 minutes and it's done.
> 
> This is assuming there are no other obstacles aside from the
> mechanics of installing changes into our repository.

I will again say that I don't expect that my situation is typical.
There may be no one else out there who ever prefers something like it.

But FWIW I think that what you describe would work for me.

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On 11 Nov 2015 11:27 pm, "Richard Stallman" <rms@gnu.org> wrote:
>
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.

Richard, do think such as system would be viable copyright-wise?
Of course, we would only do this when the author assigns copyright, but
there's a very real possibility that a careless maintainer might add
someone else's code to this page. And then this (non-copyright-assigned)
code would be automatically pulled and committed to Elpa.

I ask this because if we are OK with this risk then I'm perfectly willing
to write such a script (eager, even).

[-- Attachment #2: Type: text/html, Size: 949 bytes --]

Re: ELPA policy

[Dmitry Gutov]

On 11/12/2015 02:35 AM, Artur Malabarba wrote:

> Richard, do think such as system would be viable copyright-wise?
> Of course, we would only do this when the author assigns copyright, but
> there's a very real possibility that a careless maintainer might add
> someone else's code to this page. And then this (non-copyright-assigned)
> code would be automatically pulled and committed to Elpa.

We should also consider whether this is sound from the security 
standpoint. I mean, pushing the code to ELPA without any review from the 
core developers.

Re: ELPA policy

[Stephen Leake]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Since pulling in data from the Web indiscriminately is not
>   > possible for this project (we have to make sure we're not
>   > unwittingly including code without a proper copyright assignment),
>   > perhaps we need as an "integrator": someone willing to guide the
>   > update of ELPA from various sources on the Web, keeping an eye out
>   > for changes that might affect copyright.
>
> What we need to solve Drew's problem is to merge changes into a few
> specific files from a specific place, and commit them using git.
> Isn't that easy enough to do?
>
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.
>
> Thus, when Drew wants to install a new version, he would change the
> other pages first, then write the change log text into the log page.
> Then wait 5 minutes and it's done.
>
> This is assuming there are no other obstacles aside from the
> mechanics of installing changes into our repository.

Any malicious hacker can drop completely different code in that web
page, and thus get it into Gnu ELPA.

We will have replaced the security of private machines with whatever web
login that web page requires; that's a huge step backwards.

-- 
-- Stephe

Re: xref and GNU Global (Re: ELPA policy)

[Stephen Leake]

Nicolas Petton <nicolas@petton.fr> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>>> What about having support for Global (I haven't followed the development
>>> of xref in a while, so I might have missed it).  I'm using ggtags now,
>>> but having support for it in xref would be awesome.
>>
>> I'm working on it, but I seem to be going very slowly.
>
> Do you want help?

Yes, but the main reason I'm not making progress on this backend is that
I keep getting distracted by other project.el and xref issues.

Here's what I have so far - feel free to play with it:

--- file symref-patches.el
;; Additions to semantic/symref.el

;; semantic/symref implementation of xref-find-function

(defun semantic-symref--xref-find-definitions (symbol &optional symref-tool)
  "Implement `xref-find-function' for 'definitions for the semantic/symref backend.
SYMREF-TOOL determines which symref tool to use (default
'detect); see `semantic-symref-tool-alist'."
  (require 'semantic/symref)
  (defvar semantic-symref-tool)
  (let* ((semantic-symref-tool (or symref-tool 'detect))
         (res (semantic-symref-find-tags-by-name symbol 'project))
         (hits (and res (oref res hit-lines)))
	 xrefs)

    (dolist (hit hits)
      (push
       (xref-file-location
	:file (cdr hit)
	:line (car hit)
	:column 0)
       ;; FIXME: add :hint (match-string 3))
       xrefs))

    xrefs))

(defun semantic-symref-xref-find (action id)
  "Implement `xref-find-function' for semantic/symref."
  (cl-ecase action
    (definitions
      (semantic-symref--xref-find-definitions id))
    (references
     (semantic-symref--xref-find-references id))
    (apropos
     (semantic-symref--xref-find-apropos id)))
  )

(provide 'symref-patches)
--- end file


> BTW, regarding CEDET, I never tried it, but I don't plan to use it
> either.  I think the best would be to just add a simple backend for
> xref, with no dependency to CEDET.

The above requires that you enable EDE via:

 (require 'ede)
 (global-ede-mode)
 (ede-enable-generic-projects)

but it does not require semantic-mode.

EDE is required because the symref backend detection uses EDE to find
the project root. For Gnu Global, that could be changed to one call to
locate-dominating-file. But I'm waiting for the current CEDET merge to
master to be complete before suggesting changes to this.

This gets to the issue of whether CEDET is part of the Emacs standard
library or not. If it is, we should use it, while also making it more
transparent to users like you, who would rather not be bothered by it.
Which is an example of the kind of thing I get distracted by :).

Can you elaborate on why you don't want to use CEDET?

I'm assuming you mean you don't want to enable semantic-mode or
ede-mode; you should not care whether the file cedet-global.el is
loaded. 

-- 
-- Stephe

Re: xref and GNU Global (Re: ELPA policy)

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/12/2015 01:19 AM, Stephen Leake wrote:
>
>> There's no reason to have both cedet-global and ggtags.
>
> ggtags has some other things as well, like a
> completion-at-point-functions element, hooks to update the current
> database automatically, eldoc support, etc.

Right. All of which should also be in CEDET.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

Stephen,

On 11/11/2015 15:22 -0600, Stephen Leake wrote:

>> If sufficient amount of important packages use some API and that API
>> is rather mature, then the maintainer can decide to move that in core
>> to simplify dependency management.
>
> or to a tarball ELPA package, or to a core ELPA package.
>
> Perhaps that is too much choice?

Mm yes, that's one of my fears, that it will become too complex for a
package maintaner.  Why not just treat each ELPA package just as an ELPA
package and leave the option of bundling it to core maintainers which
are better in this area (they'll do it in agreement with package
maintainer, of course)?

A "tarball" ELPA package is a one thing (that's what I call "bundle into
tarball", if I understood right), but "core" ELPA package is another -
here I have another fear, that normal and tarball ELPA packages
depending on such "core" packages, will have to accurately define
versions of the core package they support, and then we have to check all
this during the installation.  We'll probably have to calculate and
offer to user which set of the multiple core packages' multiple versions
suits for multiple normal and tarball (perhaps overriden by version from
Internet package archive) packages, at the same time probably giving the
user to opportunity to specify her preferred version of each requested
package.  Is it worth the trouble?  Or do I understand something wrong?

That's why I wrote about the feature latency - maybe it would be enough
if a person willing to use latest core features in her package will have
to develop it on git emacs and wait for the next official release to
make her package available to the public.  This will be the same as with
new C level features, which we cannot "push quicky" as we can with ELPA
package.

>> So, I'd suggest that:
>>
>> - Language features go straight into core (and people working on them
>>   / using them will have to use git version of Emacs until next
>>   release)
>
> By "language features" do you mean things like ada-mode.el (primary mode
> for editing Ada source files)? If so, I strongly disagree; ada-mode.el
> is very happy as a normal ELPA package. Similar modes for other
> languages with a small audience could be in ELPA as well; I have not
> looked to see what's actually there.

No-no, what I meant were Emacs Lisp libraries extending language, like
seq.el.

>> The API / SPI notion can also be used to provide implementations
>> (backends) for different features which may have default simple
>> implementations in core and more advanced ones in packages.  A package
>> must somehow "register" itself as a candidate for being a service for
>> a concrete feature during installation.
>
> If we use cl-generic to provide dispatching, the cl-defmethod form does
> the registration.

Thanks, that's one more reason for me to become acquainted with Emacs CL
features.

Filipp

Re: ELPA policy

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>> If a package is useful to recommend, and we CAN get papers for it, we
>> definitely want to move it to GNU ELPA.
>
> Yes, this is why I want to clearly define ELPA policy, and streamline things
> as much as possible for developers and users: so that it becomes a more
> attractive means for distributing Emacs packages. I think many people may be
> largely ignoring it right now, and so they reach for MELPA. Since so many
> people contribute to MELPA, they consider it a more attractive distribution
> platform.


MELPA is also *easier* to contribute to. Aside from copyright issues, it
involves writing something that looks like lisp, testing on your local
fork, then a PR.

With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
could be made easier by allowing recipes, and by accepting PRs. This
avoids the necessity to have commit access to GNU ELPA to contribute.

Finally, I think that core devs should contribute to individual packages
by PR to their repos.

Phil

RE: ELPA policy

[Drew Adams]

> Any malicious hacker can drop completely different code in that web
> page, and thus get it into Gnu ELPA.  We will have replaced the
> security of private machines with whatever web login that web page
> requires; that's a huge step backwards.

Again: "I post my code to Emacs Wiki (to locked pages)."

But of course that doesn't mean it is uncrackable.

Re: ELPA policy

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
> could be made easier by allowing recipes, and by accepting PRs. This avoids
> the necessity to have commit access to GNU ELPA to contribute.

We can always use the power of Git here: Keep a fork of ELPA.git on GitHub,
accept PRs there, and periodically merge back in.

John

Re: ELPA policy

[John Wiegley]

>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:

> A "tarball" ELPA package is a one thing (that's what I call "bundle into
> tarball", if I understood right), but "core" ELPA package is another

I must have missed this distinction. What is the difference between "tarball
ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
Emacs.git.

John

Re: ELPA policy

[Alex Schröder]

Drew Adams <drew.adams@oracle.com> writes:

> Again: "I post my code to Emacs Wiki (to locked pages)."
>
> But of course that doesn't mean it is uncrackable.

Given that Emacs Wiki currently does not use HTTPS, we should consider
Emacs Wiki insecure. Even then, the admin passwords are shared and there
are no other efforts made to identify editors once they present the
admin password. The current setup only protects against the simplest of
threat models, e.g. spammers, vandals, well-meaning contributors that
want to edit wiki pages instead of informing maintainers, and so on. I
think that's the threat model wiki pages need to defend against, but I
suspect that's not good enough for an inclusion into Emacs Core.

-- 
Public Key Fingerprint = DF94 46EB 7B78 4638 7CCC  018B C78C A29B ACEC FEAE

Re: ELPA policy

[Filipp Gunbin]

On 12/11/2015 09:11 -0800, John Wiegley wrote:

>>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>> tarball", if I understood right), but "core" ELPA package is another
>
> I must have missed this distinction. What is the difference between "tarball
> ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
> Emacs.git.

If I got it right, a "tarball" package is copied from elpa.git into the
tarball at release time.  This is to bundle package so that Internet is
not required to install it (whether or not it is installed by default).

"core" package is developed in emacs.git (because core code depends on
it and we have to have full environment in the repo), but then can be
updated from elpa if new version is available.  This is to allow
frequent updates of such packages.

If I didn't get it right, then probably other package maintainers would
feel a bit lost, too, unless we provide concise explanation of this
separation.

Filipp

Re: ELPA policy

[John Wiegley]

>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:

> "core" package is developed in emacs.git (because core code depends on it
> and we have to have full environment in the repo), but then can be updated
> from elpa if new version is available. This is to allow frequent updates of
> such packages.

Ah, that makes perfect sense, thanks.

John

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Stephen,
>
> On 11/11/2015 15:22 -0600, Stephen Leake wrote:
>
>>> If sufficient amount of important packages use some API and that API
>>> is rather mature, then the maintainer can decide to move that in core
>>> to simplify dependency management.
>>
>> or to a tarball ELPA package, or to a core ELPA package.
>>
>> Perhaps that is too much choice?
>
> Mm yes, that's one of my fears, that it will become too complex for a
> package maintaner.  Why not just treat each ELPA package just as an ELPA
> package and leave the option of bundling it to core maintainers which
> are better in this area (they'll do it in agreement with package
> maintainer, of course)?
>
> A "tarball" ELPA package is a one thing (that's what I call "bundle into
> tarball", if I understood right), 

Yes. We have not implemented this yet, but I'm imagining there is a list
of these in the Gnu Emacs Makefile somewhere (probably in a separate
file read by the Makefile). I don't think there will need to be any
metadata in the package files for this.

Declaring an ELPA package to be a tarball package does impose some
restrictions on the package maintainer; they have to synchronize with
each Emacs release, and accept the same review/oversight as core code.

> but "core" ELPA package is another -
> here I have another fear, that normal and tarball ELPA packages
> depending on such "core" packages, will have to accurately define
> versions of the core package they support, and then we have to check all
> this during the installation.  

I don't see why that is different from depending on a normal ELPA
package. 

> We'll probably have to calculate and offer to user which set of the
> multiple core packages' multiple versions suits for multiple normal
> and tarball (perhaps overriden by version from Internet package
> archive) packages, at the same time probably giving the user to
> opportunity to specify her preferred version of each requested
> package. Is it worth the trouble? Or do I understand something wrong?

Emacs does not support multiple versions of packages available
simultaneously; there is only one instance of a package that is first in
load-path.

You can end up with one copy in the installed Emacs distribution, and
one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
the only one that other packages have to worry about. It either meets
their dependency requirement, or not.

> That's why I wrote about the feature latency - maybe it would be enough
> if a person willing to use latest core features in her package will have
> to develop it on git emacs and wait for the next official release to
> make her package available to the public.  This will be the same as with
> new C level features, which we cannot "push quicky" as we can with ELPA
> package.

That's the main reason to make a package available in both core and ELPA:
users of the released version of Emacs can use the latest version of the
core ELPA package from ELPA, overriding the copy in their installation.

>>> So, I'd suggest that:
>>>
>>> - Language features go straight into core (and people working on them
>>>   / using them will have to use git version of Emacs until next
>>>   release)
>
> No-no, what I meant were Emacs Lisp libraries extending language, like
> seq.el.

That's a good candidate for a core ELPA package.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>>>>>>> Richard Stallman <rms@gnu.org> writes:
>>
>>> If a package is useful to recommend, and we CAN get papers for it, we
>>> definitely want to move it to GNU ELPA.
>>
>> Yes, this is why I want to clearly define ELPA policy, and streamline things
>> as much as possible for developers and users: so that it becomes a more
>> attractive means for distributing Emacs packages. I think many people may be
>> largely ignoring it right now, and so they reach for MELPA. Since so many
>> people contribute to MELPA, they consider it a more attractive distribution
>> platform.
>
>
> MELPA is also *easier* to contribute to. Aside from copyright issues, it
> involves writing something that looks like lisp, testing on your local
> fork, then a PR.
>
> With GNU ELPA, it involves some fairly obscure git cleverness. 

It's only normal git cleverness; there's nothing special about the Gnu
ELPA git. Unless you are using an "external" package, perhaps.

I can understand treating all of git as "obscure", if that's what you
meant. I much prefer monotone.

> GNU ELPA could be made easier by allowing recipes, and by accepting
> PRs. This avoids the necessity to have commit access to GNU ELPA to
> contribute.

"PR" is "Pull Request"? Doesn't that mean "pull from my git repository"?
Or is it more general than that?

> Finally, I think that core devs should contribute to individual packages
> by PR to their repos.

I'm guessing "their repos" is _not_ Gnu ELPA git? So this only applies
to packages that have a primary repo that is not Gnu ELPA git.

I agree that any one that edits a Gnu ELPA package in Gnu ELPA git
should also notify the upstream project if there is one. But they do not
have to wait for approval if it's a critical bug.

Normal ettiquette should apply of course.

This is more important for tarball and core ELPA packages, since they
are part of the blessed Emacs standard library; Emacs developers can
edit those as if they were in core.

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Any malicious hacker can drop completely different code in that web
  > page, and thus get it into Gnu ELPA.

Drew said the pages were locked.
Doesn't that mean that only he has access to change them?

  > We will have replaced the security of private machines with whatever web
  > login that web page requires; that's a huge step backwards.

I think you are concerned that someone might break the security on that other
server and then install changes on it using Drew's account.

In general, someone who breaks the security on a machine used by
an Emacs contributor might be able to insert changes in Emacs
by pretending to be that contributor.  I don't think this is
fundamentally different.  But maybe the web site's security is
not quite as good.

We can make the security tighter.  Drew, are you willing to GPG-sign
your new versions?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, do think such as system would be viable copyright-wise?

In terms of copyright, there is no difference between the system I
proposed (installing what Drew puts in certain web pages) and the
method Drew doesn't like, which is for him to install the same changes
via Git.

Whichever method he uses, he technically could install code written by
others whom we don't have papers for.  If we permit him to use Git to
install changes, that entails trusting him to handle this
responsibility.  If we can trust him to do it with one method, we can
trust him to do it with the other method.

  > We should also consider whether this is sound from the security 
  > standpoint. I mean, pushing the code to ELPA without any review from the 
  > core developers.

The same would occur if we invite him to install changes in his packages
through Git.

If we can check enough one way, we can check enough the other way.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: ELPA policy

[Drew Adams]

>   > Any malicious hacker can drop completely different code in that web
>   > page, and thus get it into Gnu ELPA.
> 
> Drew said the pages were locked.
> Doesn't that mean that only he has access to change them?

No, anyone with admin privileges for the wiki has access to do so.
There are a few people in this category.  And see Alex Schroeder's
clarification of what this means.  This is not watertight security,
by any means.

Perhaps one way to look at it is similar to submitting something
by email (which would be another possibility, for me).

>   > We will have replaced the security of private machines with whatever
>   > web login that web page requires; that's a huge step backwards.
> 
> I think you are concerned that someone might break the security on that
> other server and then install changes on it using Drew's account.

See above.

> In general, someone who breaks the security on a machine used by
> an Emacs contributor might be able to insert changes in Emacs
> by pretending to be that contributor.  I don't think this is
> fundamentally different.  But maybe the web site's security is
> not quite as good.
> 
> We can make the security tighter.  Drew, are you willing to GPG-sign
> your new versions?

I don't really know what that entails.

Dunno whether you really want to discuss my case in particular in
detail here.  Again, I doubt that it is typical.  The reason for
my initial message about this was to suggest that some people do
use MELPA, and that perhaps some way to accommodate them could be
devised.  But maybe not.

To repeat the summary of my original point:

  So you might recommend that packages not be put in MELPA, but
  some will continue to be put there, including perhaps some that
  you might someday want to include in Emacs.

Regarding my own case, this was the point:

  I do not use GIT, so any updates I make to them
  would not be done directly in the repository.  It was not
  acceptable to update elsewhere (e.g. the wiki) and then have
  someone or a program pull from there to the repository when
  appropriate.

In sum, some people will post code to MELPA, including some
that you might someday want in Emacs.  And some input to MELPA
comes from the wiki, not from GIT - but this is probably a small
portion of what is in MELPA.

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On 12 Nov 2015 10:34 pm, "Richard Stallman" <rms@gnu.org> wrote:
>   > Richard, do think such as system would be viable copyright-wise?
>
> In terms of copyright, there is no difference between the system I
> proposed (installing what Drew puts in certain web pages) and the
> method Drew doesn't like, which is for him to install the same changes
> via Git.
>
> Whichever method he uses, he technically could install code written by
> others whom we don't have papers for.  If we permit him to use Git to
> install changes, that entails trusting him to handle this
> responsibility.  If we can trust him to do it with one method, we can
> trust him to do it with the other method.

I'm trying to determine whether this method can be employed as a general
elpa mechanism. Of course we trust Drew, but would it be acceptable to use
such a script with other people who submit packages to Elpa? (assuming the
author has assigned copyright)

[-- Attachment #2: Type: text/html, Size: 1135 bytes --]

Re: ELPA policy

[Eli Zaretskii]

> Date: Thu, 12 Nov 2015 15:05:50 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: jwiegley@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> 
> Regarding my own case, this was the point:
> 
>   I do not use GIT, so any updates I make to them
>   would not be done directly in the repository.  It was not
>   acceptable to update elsewhere (e.g. the wiki) and then have
>   someone or a program pull from there to the repository when
>   appropriate.
> 
> In sum, some people will post code to MELPA, including some
> that you might someday want in Emacs.  And some input to MELPA
> comes from the wiki, not from GIT - but this is probably a small
> portion of what is in MELPA.

FWIW, I very much doubt that many people nowadays would share Drew's
issue with using Git.  Using Git is a no-brainer nowadays.  I actually
think this is a very unique situation, and anyone else who insists on
using MELPA doesn't do that for such simple technical reasons.  I
think the legal paperwork is the main issue, and also perhaps not
everyone would like to be accountable for their code practices before
the Emacs maintainers and their peculiar requirements and
expectations.

Re: ELPA policy

[Filipp Gunbin]

On 12/11/2015 13:52 -0600, Stephen Leake wrote:

>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>> tarball", if I understood right), 
>
> Yes. We have not implemented this yet, but I'm imagining there is a list
> of these in the Gnu Emacs Makefile somewhere (probably in a separate
> file read by the Makefile). I don't think there will need to be any
> metadata in the package files for this.
>
> Declaring an ELPA package to be a tarball package does impose some
> restrictions on the package maintainer; they have to synchronize with
> each Emacs release, and accept the same review/oversight as core code.

They'll just have to make sure that a single version (which is going to
be bundled in a tarball) works good with an Emacs release being
prepared.

>> but "core" ELPA package is another -
>> here I have another fear, that normal and tarball ELPA packages
>> depending on such "core" packages, will have to accurately define
>> versions of the core package they support, and then we have to check all
>> this during the installation.  
>
> I don't see why that is different from depending on a normal ELPA
> package.

I think it's their dependants what make them different from tarball and
normal packages.

Emacs core provides API, which others use.  A normal package declares
which versions of this API it is supposed to work against.

With core packages, Emacs still provides API, but it now consists of a
static part (C level & core) and dynamic part (core packages, which can
later be updated from ELPA - correct me if I'm wrong).  So, a package
should independently define which core version it works agains and which
core package(s) version(s) it works against.

Here's where I see the complexity with multiple packages installed on
user request with package manager, which I tried to describe below.

>> We'll probably have to calculate and offer to user which set of the
>> multiple core packages' multiple versions suits for multiple normal
>> and tarball (perhaps overriden by version from Internet package
>> archive) packages, at the same time probably giving the user to
>> opportunity to specify her preferred version of each requested
>> package. Is it worth the trouble? Or do I understand something wrong?
>
> Emacs does not support multiple versions of packages available
> simultaneously; there is only one instance of a package that is first in
> load-path.
>
> You can end up with one copy in the installed Emacs distribution, and
> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
> the only one that other packages have to worry about. It either meets
> their dependency requirement, or not.

Of course, I was talking about the set of available versions which could
be installed and from which a user or a package manager should choose.

>> That's why I wrote about the feature latency - maybe it would be enough
>> if a person willing to use latest core features in her package will have
>> to develop it on git emacs and wait for the next official release to
>> make her package available to the public.  This will be the same as with
>> new C level features, which we cannot "push quicky" as we can with ELPA
>> package.
>
> That's the main reason to make a package available in both core and ELPA:
> users of the released version of Emacs can use the latest version of the
> core ELPA package from ELPA, overriding the copy in their installation.

Yes, but it can introduce complexities I wrote of above.

Filipp

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > MELPA is also *easier* to contribute to. Aside from copyright issues, it
  > involves writing something that looks like lisp, testing on your local
  > fork, then a PR.

What is a PR?

  > With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
  > could be made easier by allowing recipes, and by accepting PRs. This
  > avoids the necessity to have commit access to GNU ELPA to contribute.

Maybe we should do this (but I don't know what a PR is).

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We can make the security tighter.  Drew, are you willing to GPG-sign
  > > your new versions?

  > I don't really know what that entails.

It means you would make diffs to show the change to install,
plus a log entry, sign that whole thing with GnuPG, and mail
it to a certain address.  The signature would demonstrate it came
from you.

Is that ok for you?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: ELPA policy

[Drew Adams]

>   > > We can make the security tighter.  Drew, are you willing
>   > > to GPG-sign your new versions?
> 
>   > I don't really know what that entails.
> 
> It means you would make diffs to show the change to install,
> plus a log entry, sign that whole thing with GnuPG, and mail
> it to a certain address.  The signature would demonstrate it
> came from you.
> 
> Is that ok for you?

I have no problem creating diffs & log entries, and emailing
them in.  I've mailed in plenty of patches that way.

Dunno what "sign that whole thing with GnuPG" entails.  What
is needed for that (e.g., on MS Windows)?

Looking at the doc at https://www.gnupg.org/documentation/howtos.html
and http://www.dewinter.com/gnupg_howto/english/GPGMiniHowto-2.html,
it seems I would need to install GnuPG from source code,
using a C compiler etc.  I don't plan to install and use tools
such as a C compiler on my laptop, so I guess the answer would
be no; sorry.

If there were an easy-to-use Windows binary for such signing,
then I would probably do as you propose.

(I don't do software development anymore.  I use MS Windows
binaries of Emacs, and I write some Emacs-Lisp code.  I do
have (an old version of) Cygwin installed, so that I can use
a few simple commands from within Emacs.  That's about it.)

I don't mean to make a big deal about this.  I'm not saying
no definitively.  But AFAIK there is not necessarily even
any interest in adding any of the code I've written to Emacs.
I don't feel like going through a lot of preparation for
nothing.

My libraries are there, and have been for quite a while.
If someone is really interested in something, let me know
and maybe we can work something out.  Otherwise, we are
likely wasting each other's time here.  I don't mean to
waste your time.  (But I do appreciate your having
considered this.)

Again, my original point was about MELPA and updates to
it from the wiki.  And again, my use of MELPA is likely
not typical.  Perhaps I should have said nothing at all
about it.

Re: ELPA policy

[JJ Asghar]

On 11/13/15 3:58 PM, Richard Stallman wrote:
>   > MELPA is also *easier* to contribute to. Aside from copyright issues, it
>   > involves writing something that looks like lisp, testing on your local
>   > fork, then a PR.
> 
> What is a PR?

A great detailed explanation of a PR is here[1]. A short explanation is
a request to put in a patch to a repository.

[1]: https://help.github.com/articles/using-pull-requests/

-JJ

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Dunno what "sign that whole thing with GnuPG" entails.  What
  > is needed for that (e.g., on MS Windows)?

GnuPG works on Windows.  I expect we can find someone to help
you install it.  Then we need you to send us your public key
so we can verify your signatures.  We can find a way to do that.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Achim Gratz]

Drew Adams writes:
> Dunno what "sign that whole thing with GnuPG" entails.  What
> is needed for that (e.g., on MS Windows)?

http://www.gpg4win.org/

> (I don't do software development anymore.  I use MS Windows
> binaries of Emacs, and I write some Emacs-Lisp code.  I do
> have (an old version of) Cygwin installed, so that I can use
> a few simple commands from within Emacs.  That's about it.)

You'd really be off easier if you updated Cygwin so that you could use
Git and SSH and GPG from that.  Perhaps you'll find that using Emacs
from Cygwin is a lot easier now, too -- you can use emacs-w32 if you
don't care about running X11.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 12/11/2015 09:11 -0800, John Wiegley wrote:
>
>>>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>
>>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>>> tarball", if I understood right), but "core" ELPA package is another
>>
>> I must have missed this distinction. What is the difference between "tarball
>> ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
>> Emacs.git.
>
> If I got it right, a "tarball" package is copied from elpa.git into the
> tarball at release time.  This is to bundle package so that Internet is
> not required to install it (whether or not it is installed by default).
>
> "core" package is developed in emacs.git (because core code depends on
> it and we have to have full environment in the repo), but then can be
> updated from elpa if new version is available.  This is to allow
> frequent updates of such packages.

Yes.

I'm working on writing this up for admin/notes/elpa.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 12/11/2015 13:52 -0600, Stephen Leake wrote:
>
>> I don't see why that is different from depending on a normal ELPA
>> package.
>
> I think it's their dependants what make them different from tarball and
> normal packages.
>
> Emacs core provides API, which others use.  A normal package declares
> which versions of this API it is supposed to work against.
>
> With core packages, Emacs still provides API, but it now consists of a
> static part (C level & core) and dynamic part (core packages, which can
> later be updated from ELPA - correct me if I'm wrong).  So, a package
> should independently define which core version it works agains and which
> core package(s) version(s) it works against.

package.el dependencies can only specify a minimum version, not a
maximum. there is no way that a normal ELPA package can declare that it
works with seq.el 1.0 but not seq.el 1.1

So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration

;; Package-Requires: ((emacs "25.1"))

is equivalent to:

;; Package-Requires: ((emacs "25.1") (seq "1.0))

If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
in either case.

Listing a core ELPA package explicitly is only necessary when the emacs
version is less than the first version that included the core ELPA
package version.

On the other hand, it doesn't hurt, so it's probably best to recommend
listing all ELPA packages explicitly.

>>> We'll probably have to calculate and offer to user which set of the
>>> multiple core packages' multiple versions suits for multiple normal
>>> and tarball (perhaps overriden by version from Internet package
>>> archive) packages, at the same time probably giving the user to
>>> opportunity to specify her preferred version of each requested
>>> package. Is it worth the trouble? Or do I understand something wrong?
>>
>> Emacs does not support multiple versions of packages available
>> simultaneously; there is only one instance of a package that is first in
>> load-path.
>>
>> You can end up with one copy in the installed Emacs distribution, and
>> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
>> the only one that other packages have to worry about. It either meets
>> their dependency requirement, or not.
>
> Of course, I was talking about the set of available versions which could
> be installed and from which a user or a package manager should choose.

I still don't see a problem.

We have an example of a core ELPA package now; ada-mode. version 4.3 is
in emacs git; version 5.1.8 is in ELPA (if I ever get 5.x to be fast
enough on huge files, I'll delete 4.3 from core).

package.el has no issues with this.

Similar things can occur when there are different versions of the same
package in multiple repositories. In that sense, emacs git is just
another package repository.

Can you be more explicit about what problem you foresee? Give an example
of a package that would cause a problem.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Richard Stallman <rms@gnu.org> writes:

>   > We will have replaced the security of private machines with whatever web
>   > login that web page requires; that's a huge step backwards.
>
> I think you are concerned that someone might break the security on that other
> server and then install changes on it using Drew's account.
>
> In general, someone who breaks the security on a machine used by
> an Emacs contributor might be able to insert changes in Emacs
> by pretending to be that contributor.  I don't think this is
> fundamentally different.  But maybe the web site's security is
> not quite as good.

Yes, my concern is about the strength of the web security, as opposed to
the strength of Drew's private machine.

I suppose there's no a priori reason to assume one is more secure than
the other; maybe I'm just reacting to all the news reports about hacking.

-- 
-- Stephe

Re: ELPA policy

[Jorge A. Alfaro-Murillo]

Stephen Leake writes:
 
> Including a package in tarball ELPA means that the Emacs project 
> has decided that the package is sufficiently important to be 
> made immediately available to all Emacs users with no effort on 
> their part beyond installing Emacs from the tarball. 
> 
> That implies a certain level of maturity and quality, and 
> implies that it should be favored over other similar packages.

Would that just be packages that are currently in core? Because 
AUCTeX definitely fits that description. I have always been 
curious as why it is not in core currently.

Best,
-- 
Jorge.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I suppose there's no a priori reason to assume one is more secure than
  > the other; maybe I'm just reacting to all the news reports about hacking.

Would you please call that "cracking"?  Hacking is what we do as Emacs
developers.  See http://stallman.org/articles/on-hacking.html.

Anyway, now I understand your concern, and I think it's a valid one.
That's why I've proposed another solution.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Michael Heerdegen]

Richard Stallman <rms@gnu.org> writes:

> GnuPG works on Windows.  I expect we can find someone to help
> you install it.  Then we need you to send us your public key
> so we can verify your signatures.  We can find a way to do that.

I think that's not much easier for Drew than to use git.

I know Drew doesn't want to use git in his workflow (which he doesn't
have to), he does not dislike git for political reasons or so AFAIR.

Once he has set up things correctly (we can help him with that), he just
has to change the file in his repo clone, pull, add and push.  That's not
more complicated than what he does now when uploading to the wiki.  He
doesn't have to use git for anything else, or while developing.

Does that sound acceptable to you, Drew?  I can help you with any
problems you might encounter.  After everything has been set up
properly, uploading to Gnu Elpa is trivial, you don't have to learn git
for that.


Regards,

Michael.

RE: ELPA policy

[Drew Adams]

> Once he has set up things correctly (we can help him with that), he
> just has to change the file in his repo clone, pull, add and push.  
> That's not more complicated than what he does now when uploading to
> the wiki.  He doesn't have to use git for anything else, or while 
> developing.
> 
> Does that sound acceptable to you, Drew?  I can help you with any
> problems you might encounter.  After everything has been set up
> properly, uploading to Gnu Elpa is trivial, you don't have to
> learn git for that.

I will take a look.  Please follow up with me off-list.

(I've tried to make clear that my reply to this thread was
not about getting my libraries into Emacs, but was about
"MELPA and updates to it from the wiki."  And as I said,
"AFAIK there is not necessarily even any interest in adding
any of the code I've written to Emacs."  This has largely
been much ado about nothing, I think.  I'm sorry for the
resulting waste of people's time.  But I'm glad that the
question of MELPA and wiki code was at least considered
to some extent.)

Re: ELPA policy

[John Wiegley]

>>>>> Jorge A Alfaro-Murillo <jorge.alfaro-murillo@yale.edu> writes:

>> Including a package in tarball ELPA means that the Emacs project has
>> decided that the package is sufficiently important to be made immediately
>> available to all Emacs users with no effort on their part beyond installing
>> Emacs from the tarball.
>> 
>> That implies a certain level of maturity and quality, and implies that it
>> should be favored over other similar packages.

> Would that just be packages that are currently in core? Because AUCTeX
> definitely fits that description. I have always been curious as why it is
> not in core currently.

No, it wouldn't be just packages currently in core, we could consider others
for inclusion as well. AucTeX might be an excellent candidate.

I like that ELPA could give us better flexibility about what goes in the
tarball, without having to make decisions that directly affect core.

John

Re: ELPA policy

[Filipp Gunbin]

Stephen,

On 14/11/2015 04:30 -0600, Stephen Leake wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> On 12/11/2015 13:52 -0600, Stephen Leake wrote:
>>
>>> I don't see why that is different from depending on a normal ELPA
>>> package.
>>
>> I think it's their dependants what make them different from tarball and
>> normal packages.
>>
>> Emacs core provides API, which others use.  A normal package declares
>> which versions of this API it is supposed to work against.
>>
>> With core packages, Emacs still provides API, but it now consists of a
>> static part (C level & core) and dynamic part (core packages, which can
>> later be updated from ELPA - correct me if I'm wrong).  So, a package
>> should independently define which core version it works agains and which
>> core package(s) version(s) it works against.
>
> package.el dependencies can only specify a minimum version, not a
> maximum. there is no way that a normal ELPA package can declare that it
> works with seq.el 1.0 but not seq.el 1.1

My words were more of a theoretical speculation rather than discussion
of the current state of package.el.

Some parts of API may be deprecated and removed, so it may be the case
that a current package is not updated instantly and need to use some
previous version of core package.  While having a "single snapshot" of
Emacs + core packages does not cause such issues (even if package does
not specify maximum version which author of course does not know in
advance, the package just doesn’t work and so the user can downgrade to
previous "snapshot", that is, previous Emacs version), separate core
packages update after installation of such a "snapshot" might move
forward and thus break some normal package (and downgrade will not help
- the user will not know what to downgrade - what package? or core
emacs?  And how can I downgrade a single (core) package?).  Just theory,
again.  Ignore if it is irrelevant and I’m complicating things too much.

> So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration
>
> ;; Package-Requires: ((emacs "25.1"))
>
> is equivalent to:
>
> ;; Package-Requires: ((emacs "25.1") (seq "1.0))
>
> If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
> in either case.

Will the user have the option to continue to use the tarball version
instead of newer ELPA version?  Or choosing which ELPA version to use?
This may be needed when she uses previous major Emacs release, and
current ELPA package version requires newer core APIs.

>>>> We'll probably have to calculate and offer to user which set of the
>>>> multiple core packages' multiple versions suits for multiple normal
>>>> and tarball (perhaps overriden by version from Internet package
>>>> archive) packages, at the same time probably giving the user to
>>>> opportunity to specify her preferred version of each requested
>>>> package. Is it worth the trouble? Or do I understand something wrong?
>>>
>>> Emacs does not support multiple versions of packages available
>>> simultaneously; there is only one instance of a package that is first in
>>> load-path.
>>>
>>> You can end up with one copy in the installed Emacs distribution, and
>>> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
>>> the only one that other packages have to worry about. It either meets
>>> their dependency requirement, or not.
>>
>> Of course, I was talking about the set of available versions which could
>> be installed and from which a user or a package manager should choose.
>
> I still don't see a problem.
>
> We have an example of a core ELPA package now; ada-mode. version 4.3 is
> in emacs git; version 5.1.8 is in ELPA (if I ever get 5.x to be fast
> enough on huge files, I'll delete 4.3 from core).
>
> package.el has no issues with this.
>
> Similar things can occur when there are different versions of the same
> package in multiple repositories. In that sense, emacs git is just
> another package repository.
>
> Can you be more explicit about what problem you foresee? Give an example
> of a package that would cause a problem.

For now, I don’t know of any practical examples.  But thanks for your
time on this.

Filipp

Re: ELPA policy

[Tom Tromey]

Stephen> package.el dependencies can only specify a minimum version, not a
Stephen> maximum. there is no way that a normal ELPA package can declare that it
Stephen> works with seq.el 1.0 but not seq.el 1.1

If I had to do it all over, I'd require semver for Emacs packages.
Maybe this could be done optionally somehow.

Tom

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

>> So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration
>>
>> ;; Package-Requires: ((emacs "25.1"))
>>
>> is equivalent to:
>>
>> ;; Package-Requires: ((emacs "25.1") (seq "1.0))
>>
>> If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
>> in either case.
>
> Will the user have the option to continue to use the tarball version
> instead of newer ELPA version?  

Yes, they may install it or not; my statement above assumed the ELPA
package was installed and updated.

> Or choosing which ELPA version to use?

Technically, there is only one ELPA version; the one shown by
`list-packages'. However, after user installs one version, they may opt
to not install an update.

If they delete the installed version, they can only install the latest
version.

That's all just using the commands in package.el; users can of course
save code and copy it into and out of `load-path' manually.

> This may be needed when she uses previous major Emacs release, and
> current ELPA package version requires newer core APIs.

If there is a compatible ELPA package version that is more recent than
the version bundled with that Emacs version, but older than the latest
ELPA version, yes.

But that's not supported by the current ELPA server and package.el.

So at the moment, users of older Emacsen are stuck with either the
bundled package, or the latest package.

The maintainer of the ELPA package can provide compatibility code so it
is useful with older Emacsen. But that gets harder the farther back you
go; my current limit is 24.3.

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't think this is "much ado about nothing".
The reasons that affect one person may affect others.

We want people to arrange to contribute their Emacs add-ons
to Emacs, and that suggests that whenever someone says a reason
why he didn't want to put his code in GNU ELPA, we should try
to correct it.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> We want people to arrange to contribute their Emacs add-ons to Emacs, and
> that suggests that whenever someone says a reason why he didn't want to put
> his code in GNU ELPA, we should try to correct it.

Agreed. ELPA should become a more positive experience, both for developers,
contributors and users.

John

Re: ELPA policy

[Artur Malabarba]

2015-11-17 23:17 GMT+00:00 John Wiegley <jwiegley@gmail.com>:
>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>> We want people to arrange to contribute their Emacs add-ons to Emacs, and
>> that suggests that whenever someone says a reason why he didn't want to put
>> his code in GNU ELPA, we should try to correct it.
>
> Agreed. ELPA should become a more positive experience, both for developers,
> contributors and users.

I'm deeply in favor of making it easier for people to contribute code
to Elpa, and (like I said), I'm willing to help write the code for
that.
But I think the hardest part of the problem is not being discussed here.

Whatever this method is, it needs to meet two criteria simultaneously:
1. Not commit any code to Elpa until someone with push access has OK'd
it. This is just to ensure nothing malicious is being done.
2. Be fairly automated (not completely), so that the Elpa maintainers
don't have to manually commit+push all code that gets sent to them.
This is to preserve the sanity of the maintainers. The current model
where everything is commited by us just isn't scalable.

While those two points are not mutually exclusive, they are quite
difficult to reconcile. It needs to show the maintainers a diff of the
changes and then only proceed with the commit after receiving some
very minimal interaction (a reply to an email, or the click of a
button).

Re: ELPA policy

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-11-17 23:17 GMT+00:00 John Wiegley <jwiegley@gmail.com>:
>>>>>>> Richard Stallman <rms@gnu.org> writes:
>>
>>> We want people to arrange to contribute their Emacs add-ons to Emacs, and
>>> that suggests that whenever someone says a reason why he didn't want to put
>>> his code in GNU ELPA, we should try to correct it.
>>
>> Agreed. ELPA should become a more positive experience, both for developers,
>> contributors and users.
>
> I'm deeply in favor of making it easier for people to contribute code
> to Elpa, and (like I said), I'm willing to help write the code for
> that.
> But I think the hardest part of the problem is not being discussed here.
>
> Whatever this method is, it needs to meet two criteria simultaneously:
> 1. Not commit any code to Elpa until someone with push access has OK'd
> it. This is just to ensure nothing malicious is being done.
> 2. Be fairly automated (not completely), so that the Elpa maintainers
> don't have to manually commit+push all code that gets sent to them.
> This is to preserve the sanity of the maintainers. The current model
> where everything is commited by us just isn't scalable.
>
> While those two points are not mutually exclusive, they are quite
> difficult to reconcile. It needs to show the maintainers a diff of the
> changes and then only proceed with the commit after receiving some
> very minimal interaction (a reply to an email, or the click of a
> button).

With the GNU ELPA requiring copyright assignments, I doubt that we are
going to hit scalability issues of the kind you fear.  But of course
there is nothing wrong with trying to create a contributing process that
is minimal effort for the sake of such archives where this is not the
case.

-- 
David Kastrup