Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> + - Some packages are both in ELPA and core.  Currently these are
> +   largely merged into core.  With this directory structure, we should
> +   be able to git subtree these into here from ELPA.

FWIW, "git subtree" is good for what it does, but having to use it sucks.
If at all possible, we should have one and only one place where we host
the code, so we don't need "git subtree".

For those packages that are both "in core" and "in emacs", it'd be
better to extract the relevant packages directly from the elpa
repository, or vice versa (the vice versa is already supported,
obviously).


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> + - Some packages are both in ELPA and core.  Currently these are
>> +   largely merged into core.  With this directory structure, we should
>> +   be able to git subtree these into here from ELPA.
>
> FWIW, "git subtree" is good for what it does, but having to use it sucks.
> If at all possible, we should have one and only one place where we host
> the code, so we don't need "git subtree".
>
> For those packages that are both "in core" and "in emacs", it'd be
> better to extract the relevant packages directly from the elpa
> repository, or vice versa (the vice versa is already supported,
> obviously).


I've not had any experience with git subtree, but regardless, it's only
one possibility. My proof-of-principle is agnostic to how packages get
into the build. They could be pushed there with git subtree, checkouted
of ELPA (as ELPA does with externals), or just plain copied there.

I thought git subtree might be convienient where the core package is a
slight fork of the ELPA one (as org-mode is). It may also be that we
need several solutions, depending on the package. This is one reason
that I put space for multiple subdirectories.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> I've not had any experience with git subtree, but regardless, it's only
> one possibility. My proof-of-principle is agnostic to how packages get
> into the build. They could be pushed there with git subtree, checkouted
> of ELPA (as ELPA does with externals), or just plain copied there.

Good.  Extra support to do the checkout from elpa.git would be great.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've not had any experience with git subtree, but regardless, it's only
>> one possibility. My proof-of-principle is agnostic to how packages get
>> into the build. They could be pushed there with git subtree, checkouted
>> of ELPA (as ELPA does with externals), or just plain copied there.
>
> Good.  Extra support to do the checkout from elpa.git would be great.


I've been thinking a little bit about this -- in terms of the features
that would be needed, as opposed to implementation.

So far I have the ability to:

 - include some but not all packages from ELPA
 - use a specific commit/branch in ELPA which may not be the latest
 - access both external and, erm, non-external packages from ELPA
 - potentially fork the version in ELPA

My feeling is that, if we are going to produce a tarball build with ELPA
packages this still needs to be repeatable.

I do not know how to support packages like seq.el where the version in
ELPA and in core are just different. I guess either ELPA needs to
support multiple versions (i.e. different Emacs versions download
different things), or maybe some "with-version" macros.

If you have any ideas or features you want, let me know -- either here
or add a TODO.org file to the branch!

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> FWIW, "git subtree" is good for what it does, but having to use it sucks.
SM> If at all possible, we should have one and only one place where we host
SM> the code, so we don't need "git subtree".

FWIW, I use git-subtree with hundreds of Emacs packages in order to include
them into my dot-emacs repository on GitHub (so that I can always clone safely
on other machines, even if those external repositories might someday become
unavailable).

So from experience using them heavily, I can say that it's not a hard thing to
manage, as long as you never want to make local changes directly to the
subtree.  There is a way to get those "back upstream", but that's where all
the real work is.  Just using subtree to keep an external repo up-to-date
within a local copy is trivial and easy.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> So from experience using them heavily, I can say that it's not a hard thing to
> manage, as long as you never want to make local changes directly to the
> subtree.

Exactly.  So if you have a package both in emacs.git and elpa.git,
you're going to have to choose which of those two is the "upstream" and
then be extra careful to tell people to only ever make changes on that
side, which I think will prove inconvenient.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> I do not know how to support packages like seq.el where the version in
> ELPA and in core are just different.

I wouldn't worry about it.  They don't have to be different (the
elpa.git version should be identical, except that it comes with an
extra file for older Emacsen).


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I do not know how to support packages like seq.el where the version in
>> ELPA and in core are just different.
>
> I wouldn't worry about it.  They don't have to be different (the
> elpa.git version should be identical, except that it comes with an
> extra file for older Emacsen).


Well, in this case, there would be some additional bloat in that
seq24.el would end up in seq25.el and would be never used. It's not a
big deal, though, no.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So from experience using them heavily, I can say that it's not a hard thing to
>> manage, as long as you never want to make local changes directly to the
>> subtree.
>
> Exactly.  So if you have a package both in emacs.git and elpa.git,
> you're going to have to choose which of those two is the "upstream" and
> then be extra careful to tell people to only ever make changes on that
> side, which I think will prove inconvenient.


There is git submodule and git subrepo (which is newer) also.

An even simpler solution would be for the emacs build to just check out
packages from ELPA during build (or vice versa).

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> Exactly. So if you have a package both in emacs.git and elpa.git, you're
SM> going to have to choose which of those two is the "upstream" and then be
SM> extra careful to tell people to only ever make changes on that side, which
SM> I think will prove inconvenient.

Maybe, but is it more inconvenient than the alternatives? I'm not sure what
the competing proposals are, exactly, at this point.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

SM> Exactly. So if you have a package both in emacs.git and elpa.git, you're
SM> going to have to choose which of those two is the "upstream" and then be
SM> extra careful to tell people to only ever make changes on that side, which
SM> I think will prove inconvenient.
> Maybe, but is it more inconvenient than the alternatives? I'm not sure what
> the competing proposals are, exactly, at this point.

The competing proposal is to have a checkout of elpa.git
within/alongside that of emacs.git.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> SM> Exactly. So if you have a package both in emacs.git and elpa.git, you're
> SM> going to have to choose which of those two is the "upstream" and then be
> SM> extra careful to tell people to only ever make changes on that side, which
> SM> I think will prove inconvenient.
>> Maybe, but is it more inconvenient than the alternatives? I'm not sure what
>> the competing proposals are, exactly, at this point.
>
> The competing proposal is to have a checkout of elpa.git
> within/alongside that of emacs.git.

The significant disadvantage to this is that if we want a stable release
of Emacs, all the ELPA packages in the tar ball now get tied to the
emacs release schedule, because all of the files in the checkout of
elpa.git will be at the same commit. Unless we move all the files in
ELPA git to be external branches.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

>> The competing proposal is to have a checkout of elpa.git
>> within/alongside that of emacs.git.
> The significant disadvantage to this is that if we want a stable release
> of Emacs, all the ELPA packages in the tar ball now get tied to the
> emacs release schedule, because all of the files in the checkout of
> elpa.git will be at the same commit.

Indeed.  There are factors that can minimize the impact, tho:
- packages in elpa.git should usually not be "crucial", so it's
  hopefully acceptable to bundle whichever revision was current at some
  particular point in time.  Users can easily update them afterwards.
- we'll definitely want the Emacs release to refer to a particular
  commit on elpa.git, and it's likely that this commit will be on
  a branch where we can apply some fixup when needed (hopefully rarely).

> Unless we move all the files in ELPA git to be external branches.

Of course, that's another option.  We should rework the GNU ELPA
building scripts to make this scale better, tho (such a rework would be
beneficial for many other reasons anyway).


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The competing proposal is to have a checkout of elpa.git
>>> within/alongside that of emacs.git.
>> The significant disadvantage to this is that if we want a stable release
>> of Emacs, all the ELPA packages in the tar ball now get tied to the
>> emacs release schedule, because all of the files in the checkout of
>> elpa.git will be at the same commit.
>
> Indeed.  There are factors that can minimize the impact, tho:
> - packages in elpa.git should usually not be "crucial", so it's
>   hopefully acceptable to bundle whichever revision was current at some
>   particular point in time.  Users can easily update them afterwards.
> - we'll definitely want the Emacs release to refer to a particular
>   commit on elpa.git, and it's likely that this commit will be on
>   a branch where we can apply some fixup when needed (hopefully rarely).


I fear that this has to be on a per-package basis though, since the
packages update to new release versions independently of each other. I
guess we can make ELPA generate this data, since something in the build
knows that this has happened.


>> Unless we move all the files in ELPA git to be external branches.
>
> Of course, that's another option.  We should rework the GNU ELPA
> building scripts to make this scale better, tho (such a rework would be
> beneficial for many other reasons anyway).


Taken to the extreme, we could just go the MELPA route; the ELPA repo
contains recipes, with the packages stored elsewhere (or on an external
branch).

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

>>> Unless we move all the files in ELPA git to be external branches.
>> Of course, that's another option.  We should rework the GNU ELPA
>> building scripts to make this scale better, tho (such a rework would be
>> beneficial for many other reasons anyway).
> Taken to the extreme, we could just go the MELPA route; the ELPA repo
> contains recipes, with the packages stored elsewhere (or on an external
> branch).

But that's exactly what I don't want: GNU ELPA should not be just
a distribution of third party packages.  For that, we already have MELPA
which works just fine.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> But that's exactly what I don't want: GNU ELPA should not be just a
SM> distribution of third party packages. For that, we already have MELPA
SM> which works just fine.

I agree also. When we make a release of Emacs, there should be at least enough
vetting of what's in ELPA that people can install those versions with some
degree of confidence. Otherwise, there'd be no reason for us to do anything
more than accumulate sources in a separate repository.

The idea behind "core" and "tarball" ELPA is that we're investing even more in
ELPA than we are now -- for a limited set of packages -- to the extent that
we're willing to add them to the tarball, or allow our core packages to depend
on their functionality.

There's also the possibility that we could add another category, maybe called
"community ELPA", that would just be a link to some external repository. In
that case, the only assurance we'd be providing is that we've reviewed their
copyright status, and have confidence they can be considered part of the GNU
Emacs project. But this would need periodic updating to be assured of
compliance, and I'm not sure we have that manpower.

Either way, our management of ELPA needs to be stepped up, with varying
degrees of oversight. The real question right now is how we achieve this
technically, and Git is the first of those puzzles we have to solve, followed
by the build system.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 325 bytes --]

On 2016-09-19 12:36, Stefan Monnier wrote:
> GNU ELPA should not be just a distribution of third party packages.
> For that, we already have MELPA which works just fine.

Well not quite: MELPA isn't officially endorsed, so users need to add the setup code to their .emacs before the can install packages from MELPA.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Unless we move all the files in ELPA git to be external branches.
>>> Of course, that's another option.  We should rework the GNU ELPA
>>> building scripts to make this scale better, tho (such a rework would be
>>> beneficial for many other reasons anyway).
>> Taken to the extreme, we could just go the MELPA route; the ELPA repo
>> contains recipes, with the packages stored elsewhere (or on an external
>> branch).
>
> But that's exactly what I don't want: GNU ELPA should not be just
> a distribution of third party packages.  For that, we already have MELPA
> which works just fine.


I think that the software and social organisation are different. The
packages could be in independent repositories on savannah for instance.
No real reason why these should be considered to be third party.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> SM> But that's exactly what I don't want: GNU ELPA should not be just a
> SM> distribution of third party packages. For that, we already have MELPA
> SM> which works just fine.
>
> Either way, our management of ELPA needs to be stepped up, with varying
> degrees of oversight. The real question right now is how we achieve this
> technically, and Git is the first of those puzzles we have to solve, followed
> by the build system.

This is my interest really. I want to be able to use package.el format
in core, put packages both in core and on ELPA, and not have ELPA
packages tightly tied to core release cycle. At the moment, these are
either not possible or a PITA.

As it happens, having this work nicely should also gives a technically
painless transition between a package being a third party, then ELPA,
then a core package. This strikes me as a good thing.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> This is my interest really. I want to be able to use package.el format in
PL> core, put packages both in core and on ELPA, and not have ELPA packages
PL> tightly tied to core release cycle. At the moment, these are either not
PL> possible or a PITA.

I hadn't thought of package.el and core packages.  What would the value be?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> The packages could be in independent repositories on savannah for
> instance.  No real reason why these should be considered to be
> third party.

Separate branches or separate repositories is indeed a distinction that
doesn't matter.  The location (and access rights) of the
repositor(y|ies) does.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The packages could be in independent repositories on savannah for
>> instance.  No real reason why these should be considered to be
>> third party.
>
> Separate branches or separate repositories is indeed a distinction that
> doesn't matter.  The location (and access rights) of the
> repositor(y|ies) does.


Sure, that's true. My point was, thought, that MELPA as it stands would
support the workflow of ELPA. The only difference, really, is that MELPA
requires tags to identify stable releases, which seems not a bad thing.
How this would work with lots of packages in one repo, though, I do not
know.

In terms of access rights, I am pretty sure that tools like gitosis
would allow creation of lots of repos, under one hood, with common
access rights.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> PL> This is my interest really. I want to be able to use package.el format in
> PL> core, put packages both in core and on ELPA, and not have ELPA packages
> PL> tightly tied to core release cycle. At the moment, these are either not
> PL> possible or a PITA.
>
> I hadn't thought of package.el and core packages.  What would the value be?

 1) Neatness -- package.el format puts all the files that provide one
 piece of functionality in one place.

 2) Familiarity -- most Emacs-lisp developers are familiar with
 package.el format. Tools like Cask and elpakit work with it.

 3) Sharing -- if they are in the same format, packages can easily be
 kept on ELPA and in core, and there is a clean migration path.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> Sure, that's true. My point was, thought, that MELPA as it stands would
> support the workflow of ELPA. The only difference, really, is that MELPA
> requires tags to identify stable releases, which seems not a bad thing.
> How this would work with lots of packages in one repo, though, I do not
> know.

Of course, one issue with one-repo-per-package is that in order to find
out which packages to (re)build, we either need to constantly poll all
the repositories, or we need some side-band signal.

Git is pretty good at updating all the branches of a single repository,
in comparison.


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

>> I hadn't thought of package.el and core packages.  What would the value be?

PL>  1) Neatness -- package.el format puts all the files that provide one
PL> piece of functionality in one place.

Are you suggesting a reorganization of the source code under lisp/?

PL>  2) Familiarity -- most Emacs-lisp developers are familiar with package.el
PL> format. Tools like Cask and elpakit work with it.

Why would anyone be rebuilding core libraries?  Or are you proposing this for
the Emacs developers?

PL>  3) Sharing -- if they are in the same format, packages can easily be kept
PL> on ELPA and in core, and there is a clean migration path.

This part I understand, as applying to the concept of "core ELPA" (which, once
I have some time next week, I will continue to discuss in our other thread).

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Sure, that's true. My point was, thought, that MELPA as it stands would
>> support the workflow of ELPA. The only difference, really, is that MELPA
>> requires tags to identify stable releases, which seems not a bad thing.
>> How this would work with lots of packages in one repo, though, I do not
>> know.
>
> Of course, one issue with one-repo-per-package is that in order to find
> out which packages to (re)build, we either need to constantly poll all
> the repositories, or we need some side-band signal.
>
> Git is pretty good at updating all the branches of a single repository,
> in comparison.


Yes, agreed. I would poll because it's simple. It might become
problematic when we have many thousands of packages, although MELPA does
it with 3000.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>> I hadn't thought of package.el and core packages.  What would the value be?
>
> PL>  1) Neatness -- package.el format puts all the files that provide one
> PL> piece of functionality in one place.
>
> Are you suggesting a reorganization of the source code under lisp/?

It's all documented in the readme on the branch.

I'm suggesting supplementing the core build with a directory called
"packages" which support package.el format packages. I can see no
compelling argument for rewriting the lisp directory in a single go. If
we have a packages directory working well, it might make sense for
individual packages.


> PL>  2) Familiarity -- most Emacs-lisp developers are familiar with package.el
> PL> format. Tools like Cask and elpakit work with it.
>
> Why would anyone be rebuilding core libraries?  Or are you proposing this for
> the Emacs developers?

Non core developers who become core developers, bringing their packages
with them.


> PL>  3) Sharing -- if they are in the same format, packages can easily be kept
> PL> on ELPA and in core, and there is a clean migration path.
>
> This part I understand, as applying to the concept of "core ELPA" (which, once
> I have some time next week, I will continue to discuss in our other thread).

Will look forward to it.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Mon, 26 Sep 2016 17:28:17 +0100
> Cc: emacs-devel@gnu.org
> 
> I'm suggesting supplementing the core build with a directory called
> "packages" which support package.el format packages.

Why does it have to be a separate directory?  Why cannot those
packages be downloaded into their current places, where they belong in
one of the subdirectories of lisp/?

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Mon, 26 Sep 2016 17:28:17 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> I'm suggesting supplementing the core build with a directory called
>> "packages" which support package.el format packages.
>
> Why does it have to be a separate directory?  Why cannot those
> packages be downloaded into their current places, where they belong in
> one of the subdirectories of lisp/?


The build process is different. I use package.el to compile, and to
build the autoloads and the -pkg.el file. This could be done in the lisp
directory, but I'd have to exclude this directory from the normal build
process in the lisp directory.

I'm not at all wedded to the directory structure. This is just a
proof-of-principle. It can be what ever we want. Is there a problem with
introducing a new top level?

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 27 Sep 2016 17:01:53 +0100
> 
> > Why does it have to be a separate directory?  Why cannot those
> > packages be downloaded into their current places, where they belong in
> > one of the subdirectories of lisp/?
> 
> 
> The build process is different. I use package.el to compile, and to
> build the autoloads and the -pkg.el file. This could be done in the lisp
> directory, but I'd have to exclude this directory from the normal build
> process in the lisp directory.

Can't we handle the differences during the build without separating
directories?  E.g., we already have some files that should not be
byte-compiled, and we handle it with a file-local variable.  Can't we
do something similar with this issue?

> I'm not at all wedded to the directory structure. This is just a
> proof-of-principle. It can be what ever we want. Is there a problem with
> introducing a new top level?

It's not a catastrophe, but it runs the risk of raising the annoyance
level when looking for a certain file.  Already there are some Lisp
files for which I never know in what subdirectory they should live.
Having yet another subtree will make things worse, since there's no
way of knowing up front whether a file is in core or not.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> > Why does it have to be a separate directory?  Why cannot those
>> > packages be downloaded into their current places, where they belong in
>> > one of the subdirectories of lisp/?
>> 
>> 
>> The build process is different. I use package.el to compile, and to
>> build the autoloads and the -pkg.el file. This could be done in the lisp
>> directory, but I'd have to exclude this directory from the normal build
>> process in the lisp directory.
>
> Can't we handle the differences during the build without separating
> directories?  E.g., we already have some files that should not be
> byte-compiled, and we handle it with a file-local variable.  Can't we
> do something similar with this issue?


Sure. We already, for example, have the obsolete directory which is
excluded from some of the build make processes.

But, I think it will ultimately be more complex. We will have one
subdirectory with a different package organisation from the all the
others.

>> I'm not at all wedded to the directory structure. This is just a
>> proof-of-principle. It can be what ever we want. Is there a problem with
>> introducing a new top level?
>
> It's not a catastrophe, but it runs the risk of raising the annoyance
> level when looking for a certain file.  Already there are some Lisp
> files for which I never know in what subdirectory they should live.
> Having yet another subtree will make things worse, since there's no
> way of knowing up front whether a file is in core or not.

Yes, this is probably true. Gains and losses.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Thu, 29 Sep 2016 10:00:54 +0100
> 
> >> The build process is different. I use package.el to compile, and to
> >> build the autoloads and the -pkg.el file. This could be done in the lisp
> >> directory, but I'd have to exclude this directory from the normal build
> >> process in the lisp directory.
> >
> > Can't we handle the differences during the build without separating
> > directories?  E.g., we already have some files that should not be
> > byte-compiled, and we handle it with a file-local variable.  Can't we
> > do something similar with this issue?
> 
> 
> Sure. We already, for example, have the obsolete directory which is
> excluded from some of the build make processes.
> 
> But, I think it will ultimately be more complex. We will have one
> subdirectory with a different package organisation from the all the
> others.

We should at least try, IMO, and not dismiss that possibility before
we really see the difficulties and can assess them.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> > Can't we handle the differences during the build without separating
>> > directories?  E.g., we already have some files that should not be
>> > byte-compiled, and we handle it with a file-local variable.  Can't we
>> > do something similar with this issue?
>> 
>> Sure. We already, for example, have the obsolete directory which is
>> excluded from some of the build make processes.
>> 
>> But, I think it will ultimately be more complex. We will have one
>> subdirectory with a different package organisation from the all the
>> others.

EZ> We should at least try, IMO, and not dismiss that possibility before we
EZ> really see the difficulties and can assess them.

I agree with Eli here. The more "natural" we can make the ELPA integration,
the better I think it will be, since it will allow the flexibility for either
approach, depending on what is best.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

>> Why would anyone be rebuilding core libraries? Or are you proposing this
>> for the Emacs developers?

PL> Non core developers who become core developers, bringing their packages
PL> with them.

I would like to not add any new core libraries, until the day we need to
depend upon them from other core libraries. New development, as much as
possible, should be directed to ELPA. The whole point of this new ELPA reorg
is that, by making it easy to include ELPA code in the distribution, there is
no longer any real reason to pull new code into core.  The core of what Emacs
does changes relatively little from version to version.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> I would like to not add any new core libraries, until the day we need to
> depend upon them from other core libraries. New development, as much as
> possible, should be directed to ELPA. The whole point of this new ELPA reorg
> is that, by making it easy to include ELPA code in the distribution, there
> is no longer any real reason to pull new code into core.  The core of what
> Emacs does changes relatively little from version to version.

I hope that:

1. The Lisp code distributed with Emacs will continue to include all
   that has been included before, so users do not need to use package.el
   etc. to obtain any of it.

2. All Emacs Lisp code, apart from 3rd-party libraries, will continue
   to be available in .../lisp/, so users can easily grep for it and
   use `M-x find-library', without messing with packages.

3. The source-code directory structure under .../lisp/ does not become
   any, or much, deeper.  So far, *.el, */*.el, and */*/*.el are
   sufficient.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> 1. The Lisp code distributed with Emacs will continue to include all that
> has been included before, so users do not need to use package.el etc. to
> obtain any of it.

Yes.

> 2. All Emacs Lisp code, apart from 3rd-party libraries, will continue to be
> available in .../lisp/, so users can easily grep for it and use `M-x
> find-library', without messing with packages.

Currently discussing this, but I'd like that too. Contributing via ELPA
shouldn't feel strangely different from how we did things in the past,
otherwise people will perceive a distinction between the two that we're now
trying to erase.

> 3. The source-code directory structure under .../lisp/ does not become any,
> or much, deeper. So far, *.el, */*.el, and */*/*.el are sufficient.

Same as #2.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
>>> > Can't we handle the differences during the build without separating
>>> > directories?  E.g., we already have some files that should not be
>>> > byte-compiled, and we handle it with a file-local variable.  Can't we
>>> > do something similar with this issue?
>>> 
>>> Sure. We already, for example, have the obsolete directory which is
>>> excluded from some of the build make processes.
>>> 
>>> But, I think it will ultimately be more complex. We will have one
>>> subdirectory with a different package organisation from the all the
>>> others.
>
> EZ> We should at least try, IMO, and not dismiss that possibility before we
> EZ> really see the difficulties and can assess them.
>
> I agree with Eli here. The more "natural" we can make the ELPA integration,
> the better I think it will be, since it will allow the flexibility for either
> approach, depending on what is best.

Sure, but it cannot change the fact that ELPA and Emacs core have
different organisations because Emacs has a unified (or monolithic
depending on your POV) directory structure, while ELPA has a
per-package structure.


So, we have

EMACS/lisp/blah.el
EMACS/test/lisp/blah-test.el
EMACS/doc/emacs/blah.texi

While ELPA has

ELPA/packages/blah/blah.el
ELPA/packages/blah/blah.texi
ELPA/packages/blah/test/blah-test.el

(actually, the test location for ELPA packages is not standardized, but
it should be).

So, we can either:

1) Have both of these organisations
2) Move EMACS to the ELPA organisation
3) Move ELPA to the EMACS organisation

My suggestion is 1). I suspect some packages in EMACS will prefer to
move to the ELPA organisation (org for example).

After that, where blah.el is a package in core format, and foo.el is a
package in elpa format, we can do:

a)

EMACS/lisp/blah.el
EMACS/packages/foo/foo.el

which is nice because it cleanly separates out the two layouts.

b)

EMACS/lisp/elpa/packages/foo/foo.el
EMACS/lisp/unified/blah.el

which also seperates the two layouts, but would require lots of file
moves form the current system.

Or c)

EMACS/lisp/blah.el
EMACS/lisp/elpa/packages/foo/foo.el

which keeps all the lisp files under the lisp top-level dir, does not
require file moves, but pushes the two directory structures together,
complicating the build.

For me, I think that the second b) is the ideal, but moving lots of
files to get there is not worth the effort. Hence I like option a). I
think c) will result in confusion of developers (as well as the build
system).

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>> Why would anyone be rebuilding core libraries? Or are you proposing this
>>> for the Emacs developers?
>
> PL> Non core developers who become core developers, bringing their packages
> PL> with them.
>
> I would like to not add any new core libraries, until the day we need to
> depend upon them from other core libraries. New development, as much as
> possible, should be directed to ELPA. The whole point of this new ELPA reorg
> is that, by making it easy to include ELPA code in the distribution, there is
> no longer any real reason to pull new code into core.  The core of what Emacs
> does changes relatively little from version to version.


Okay, we're just getting confused in terminology. Yes, I mean, people
who are current writing lisp which is not distributed with Emacs. So,
non-core developers who want to get their package into the tar ball.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:

>> I would like to not add any new core libraries, until the day we need to
>> depend upon them from other core libraries. New development, as much as
>> possible, should be directed to ELPA. The whole point of this new ELPA reorg
>> is that, by making it easy to include ELPA code in the distribution, there
>> is no longer any real reason to pull new code into core.  The core of what
>> Emacs does changes relatively little from version to version.
>
> I hope that:
>
> 1. The Lisp code distributed with Emacs will continue to include all
>    that has been included before, so users do not need to use package.el
>    etc. to obtain any of it.

I am suggesting that we enable the use of package.el as part of the
Emacs build to compile and generate the autoloads for packages in Emacs
core.


> 2. All Emacs Lisp code, apart from 3rd-party libraries, will continue
>    to be available in .../lisp/, so users can easily grep for it and
>    use `M-x find-library', without messing with packages.

This, I am suggesting we change, so that we have two locations for lisp
packages dependent on their file organisation. The lisp code will still
be available, though.

> 3. The source-code directory structure under .../lisp/ does not become
>    any, or much, deeper.  So far, *.el, */*.el, and */*/*.el are
>    sufficient.

I would not want to change this.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> a)
>
> EMACS/lisp/blah.el
> EMACS/packages/foo/foo.el
        ^^^^^^^^
What about elpa instead of packages?  It's 4 chars as 'lisp'.

> which is nice because it cleanly separates out the two layouts.
>
> b)
>
> EMACS/lisp/elpa/packages/foo/foo.el
                  ^^^^^^^^
elpa would suffice, I think.  Or is there a reason for the nested
packages directory?

> EMACS/lisp/unified/blah.el
>
> which also seperates the two layouts, but would require lots of file
> moves form the current system.
>
> Or c)
>
> EMACS/lisp/blah.el
> EMACS/lisp/elpa/packages/foo/foo.el
                  ^^^^^^^^
Same here.

Regards,
Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> After that, where blah.el is a package in core format, and foo.el is a
> package in elpa format, we can do:
>
> a)
>
> EMACS/lisp/blah.el
> EMACS/packages/foo/foo.el
>
> which is nice because it cleanly separates out the two layouts.
>
> b)
>
> EMACS/lisp/elpa/packages/foo/foo.el
> EMACS/lisp/unified/blah.el
>
> which also seperates the two layouts, but would require lots of file
> moves form the current system.
>
> Or c)
>
> EMACS/lisp/blah.el
> EMACS/lisp/elpa/packages/foo/foo.el
>
> which keeps all the lisp files under the lisp top-level dir, does not
> require file moves, but pushes the two directory structures together,
> complicating the build.
>

There's also d) where an elpa package would just go into it's
corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
an elpa package.  Of course, there's a chance of name clashes here, but
both GNU Emacs and GNU elpa are under the same control IIUC.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 30 Sep 2016 11:43:10 +0100
> 
> > EZ> We should at least try, IMO, and not dismiss that possibility before we
> > EZ> really see the difficulties and can assess them.
> >
> > I agree with Eli here. The more "natural" we can make the ELPA integration,
> > the better I think it will be, since it will allow the flexibility for either
> > approach, depending on what is best.
> 
> Sure, but it cannot change the fact that ELPA and Emacs core have
> different organisations because Emacs has a unified (or monolithic
> depending on your POV) directory structure, while ELPA has a
> per-package structure.
> 
> 
> So, we have
> 
> EMACS/lisp/blah.el
> EMACS/test/lisp/blah-test.el
> EMACS/doc/emacs/blah.texi
> 
> While ELPA has
> 
> ELPA/packages/blah/blah.el
> ELPA/packages/blah/blah.texi
> ELPA/packages/blah/test/blah-test.el
> 
> (actually, the test location for ELPA packages is not standardized, but
> it should be).
> 
> So, we can either:
> 
> 1) Have both of these organisations
> 2) Move EMACS to the ELPA organisation
> 3) Move ELPA to the EMACS organisation
> 
> My suggestion is 1).

My suggestion is none of the above.  We should instead try to arrange
things so that when a Emacs is built from the repository, the build
process updates any packages from ELPA that need to be updated, as
part of the build.  And when an Emacs release tarball is tarred, each
package has its files put/updated in the corresponding directory under
lisp/.  If we succeed doing this, there will be no difference between
packages in the Emacs repository and in ELPA, as far as building and
releasing Emacs is concerned.

Can we try doing that?

> After that, where blah.el is a package in core format, and foo.el is a
> package in elpa format, we can do:
> 
> a)
> 
> EMACS/lisp/blah.el
> EMACS/packages/foo/foo.el
> 
> which is nice because it cleanly separates out the two layouts.
> 
> b)
> 
> EMACS/lisp/elpa/packages/foo/foo.el
> EMACS/lisp/unified/blah.el
> 
> which also seperates the two layouts, but would require lots of file
> moves form the current system.

Why do we need to separate them?

> Or c)
> 
> EMACS/lisp/blah.el
> EMACS/lisp/elpa/packages/foo/foo.el
> 
> which keeps all the lisp files under the lisp top-level dir, does not
> require file moves, but pushes the two directory structures together,
> complicating the build.
> 
> For me, I think that the second b) is the ideal, but moving lots of
> files to get there is not worth the effort. Hence I like option a). I
> think c) will result in confusion of developers (as well as the build
> system).

For me, this is the ideal:

  EMACS/lisp/blah.el
  EMACS/lisp/foo.el (or EMACS/lisp/SOMETHING/foo.el)
  EMACS/doc/misc/foo-user.texi
  EMACS/doc/lispref/foo-elisp.texi

etc., where SOMETHING is one of the _existing_ subdirectories under
lisp/, as might be appropriate for foo.el's topic.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> CC: Eli Zaretskii <eliz@gnu.org>, <monnier@iro.umontreal.ca>,
> 	<emacs-devel@gnu.org>
> Date: Fri, 30 Sep 2016 13:56:18 +0200
> 
> phillip.lord@russet.org.uk (Phillip Lord) writes:
> 
> There's also d) where an elpa package would just go into it's
> corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
> an elpa package.

That's what I'd prefer, yes.

> Of course, there's a chance of name clashes here, but both GNU Emacs
> and GNU elpa are under the same control IIUC.

Exactly, and so we can easily make sure that no name clashes happen.

Besides, such name clashes would be bad even if ELPA wasn't under our
control, since loading a package could then load an incorrect one,
depending on how load-path is ordered.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> a)
>>
>> EMACS/lisp/blah.el
>> EMACS/packages/foo/foo.el
>         ^^^^^^^^
> What about elpa instead of packages?  It's 4 chars as 'lisp'.


There is already confusion between "ELPA" the web server, "ELPA" as in
any package server and "ELPA" as in the package.el format. I'm trying
not to make this worse.

>> which is nice because it cleanly separates out the two layouts.
>>
>> b)
>>
>> EMACS/lisp/elpa/packages/foo/foo.el
>                   ^^^^^^^^
> elpa would suffice, I think.  Or is there a reason for the nested
> packages directory?
>>
>> Or c)
>>
>> EMACS/lisp/blah.el
>> EMACS/lisp/elpa/packages/foo/foo.el
>                   ^^^^^^^^
> Same here.


Screw up. I would use EMACS/lisp/packages in both cases.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:
>> Or c)
>>
>> EMACS/lisp/blah.el
>> EMACS/lisp/elpa/packages/foo/foo.el
>>
>> which keeps all the lisp files under the lisp top-level dir, does not
>> require file moves, but pushes the two directory structures together,
>> complicating the build.
>>
>
> There's also d) where an elpa package would just go into it's
> corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
> an elpa package.  Of course, there's a chance of name clashes here, but
> both GNU Emacs and GNU elpa are under the same control IIUC.


Would require us to keep track of which packages are package.el format
and which packages are not, spread throughout multiple directories. As
well as making the build a PITA (and fragile when we forget to update
the list), it would be confusing for the developers who would have to
remember two different sets of package structures.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Fri, 30 Sep 2016 11:43:10 +0100
>> 
>> > EZ> We should at least try, IMO, and not dismiss that possibility before we
>> > EZ> really see the difficulties and can assess them.
>> >
>> > I agree with Eli here. The more "natural" we can make the ELPA integration,
>> > the better I think it will be, since it will allow the flexibility for either
>> > approach, depending on what is best.
>> 
>
> My suggestion is none of the above.  We should instead try to arrange
> things so that when a Emacs is built from the repository, the build
> process updates any packages from ELPA that need to be updated, as
> part of the build.  And when an Emacs release tarball is tarred, each
> package has its files put/updated in the corresponding directory under
> lisp/.  If we succeed doing this, there will be no difference between
> packages in the Emacs repository and in ELPA, as far as building and
> releasing Emacs is concerned.
>
> Can we try doing that?


We could do, but then it means that packages will use two different file
organisations; one for when they are in package.el format, and one where
they are not. Aside from being more work for the build, I think this
will be fragile. Just as a simple example consider this file layout:

package/foo/foo.el
package/foo/test/simple-test.el

package/bar/bar.el
package/bar/test/simple-test.el

where does simple-test.el go?

Emacs currrently has two systems for building packages -- one in core,
one in package.el format. Adding support package.el format in core means
that package developers only have to support one of these.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>>
>> There's also d) where an elpa package would just go into it's
>> corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
>> an elpa package.  Of course, there's a chance of name clashes here, but
>> both GNU Emacs and GNU elpa are under the same control IIUC.
>
> Would require us to keep track of which packages are package.el format
> and which packages are not, spread throughout multiple directories. As

It should be rather easy to have a convention that can reliably be used
to derive whether a given file or directory is in package.el format.
Maybe there's already one?  Or use a file-local variable as Eli
proposed?  So I think we get it nearly for "free".  Or what do you mean
by "keeping track of"?

> well as making the build a PITA (and fragile when we forget to update
> the list), it would be confusing for the developers who would have to
> remember two different sets of package structures.

I can't judge the build part of this, but I don't really see why it's a
PITA.  I also don't see what we would have to update additionally in
this layout that we wouldn't have to in a), b) and c).

Whether it's confusing or not -- well you will find arguments against
all proposed approaches.  I think d) is easier because it doesn't divide
elisp source code based on where the sources actually come from.

When I looked into GNU Emacs sources the first time, I very much
appreciated the IMO minimalistic directory structure.  I would try to
not loose it if I could.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>,  <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
> Date: Fri, 30 Sep 2016 13:41:54 +0100
> 
> > There's also d) where an elpa package would just go into it's
> > corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
> > an elpa package.  Of course, there's a chance of name clashes here, but
> > both GNU Emacs and GNU elpa are under the same control IIUC.
> 
> Would require us to keep track of which packages are package.el format
> and which packages are not, spread throughout multiple directories. As
> well as making the build a PITA (and fragile when we forget to update
> the list), it would be confusing for the developers who would have to
> remember two different sets of package structures.

If these difficulties are indeed real and grave, we could indeed use a
separate sub-tree.  But I don't think we should give up without
trying, without actually seeing the difficulties and trying to solve
them.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 30 Sep 2016 14:06:53 +0100
> 
> > My suggestion is none of the above.  We should instead try to arrange
> > things so that when a Emacs is built from the repository, the build
> > process updates any packages from ELPA that need to be updated, as
> > part of the build.  And when an Emacs release tarball is tarred, each
> > package has its files put/updated in the corresponding directory under
> > lisp/.  If we succeed doing this, there will be no difference between
> > packages in the Emacs repository and in ELPA, as far as building and
> > releasing Emacs is concerned.
> >
> > Can we try doing that?
> 
> We could do, but then it means that packages will use two different file
> organisations; one for when they are in package.el format, and one where
> they are not. Aside from being more work for the build, I think this
> will be fragile. Just as a simple example consider this file layout:
> 
> package/foo/foo.el
> package/foo/test/simple-test.el
> 
> package/bar/bar.el
> package/bar/test/simple-test.el
> 
> where does simple-test.el go?

These two files should be renamed to foo-simple-tests.el and
bar-simple-tests.el.

More generally, ELPA packages that are distributed as part of the
Emacs tarball will have to use our naming conventions, and those
include test files.  Right?

> Emacs currrently has two systems for building packages -- one in core,
> one in package.el format. Adding support package.el format in core means
> that package developers only have to support one of these.

What do you mean by "system for building packages"?  AFAIK, we don't
build packages in core, we just byte-compile them and extract
autoloads from them.

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> > I hope that:
> >
> > 1. The Lisp code distributed with Emacs will continue to include all
> >    that has been included before, so users do not need to use package.el
> >    etc. to obtain any of it.
> 
> I am suggesting that we enable the use of package.el as part of the
> Emacs build to compile and generate the autoloads for packages in Emacs
> core.

Sounds good (IIUC).  So you are talking not about users having to use
package.el but just the Emacs build using package.el.

> > 2. All Emacs Lisp code, apart from 3rd-party libraries, will continue
> >    to be available in .../lisp/, so users can easily grep for it and
> >    use `M-x find-library', without messing with packages.
> 
> This, I am suggesting we change, so that we have two locations for lisp
> packages dependent on their file organisation. The lisp code will still
> be available, though.

I am no doubt a minority of one, but I do not want to have to use
package.el at all, especially just to be able to use vanilla Emacs
(standard distribution).

> > 3. The source-code directory structure under .../lisp/ does not become
> >    any, or much, deeper.  So far, *.el, */*.el, and */*/*.el are
> >    sufficient.
> 
> I would not want to change this.

How does your answer here fit with your answer for #2?  I would like
to find everything under .../lisp/ still, with a simple subdir structure.

But again, I'm no doubt a minority of one.  Just saying what I prefer.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Alain Schneble <a.s@realize.ch> writes:
>>>
>>> There's also d) where an elpa package would just go into it's
>>> corresponding directory under EMACS/lisp, e.g. EMACS/lisp/org if org is
>>> an elpa package.  Of course, there's a chance of name clashes here, but
>>> both GNU Emacs and GNU elpa are under the same control IIUC.
>>
>> Would require us to keep track of which packages are package.el format
>> and which packages are not, spread throughout multiple directories. As
>
> It should be rather easy to have a convention that can reliably be used
> to derive whether a given file or directory is in package.el format.
> Maybe there's already one?

At the moment, no, there isn't (at least not until the -pkg.el file is
built).

> Or use a file-local variable as Eli proposed? So I think we get it
> nearly for "free". Or what do you mean by "keeping track of"?

Just what you think, yes. We need to distinguish between the two
formats. Putting them different directories, with different make files
is a trivial way to achieve this.



>> well as making the build a PITA (and fragile when we forget to update
>> the list), it would be confusing for the developers who would have to
>> remember two different sets of package structures.
>
> I can't judge the build part of this, but I don't really see why it's a
> PITA.  I also don't see what we would have to update additionally in
> this layout that we wouldn't have to in a), b) and c).

Just for what I say. Having the two package systems in different
top-level (or lower-lever) directories makes life easier. Copying files
from package.el format locations in core format would be possible but,
again, complex.

> Whether it's confusing or not -- well you will find arguments against
> all proposed approaches.  I think d) is easier because it doesn't divide
> elisp source code based on where the sources actually come from.

I understand that. But, unless we do something complex tests,
documentation, icons, subsidiary files and so forth will be in different
places for one style of packages than for the other.

> When I looked into GNU Emacs sources the first time, I very much
> appreciated the IMO minimalistic directory structure.  I would try to
> not loose it if I could.


Yes, I can appreciate that.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> We could do, but then it means that packages will use two different file
>> organisations; one for when they are in package.el format, and one where
>> they are not. Aside from being more work for the build, I think this
>> will be fragile. Just as a simple example consider this file layout:
>> 
>> package/foo/foo.el
>> package/foo/test/simple-test.el
>> 
>> package/bar/bar.el
>> package/bar/test/simple-test.el
>> 
>> where does simple-test.el go?
>
> These two files should be renamed to foo-simple-tests.el and
> bar-simple-tests.el.
>
> More generally, ELPA packages that are distributed as part of the
> Emacs tarball will have to use our naming conventions, and those
> include test files.  Right?

Yes. We don't have naming conventions for test files yet, and they will
need to be added.


>> Emacs currrently has two systems for building packages -- one in core,
>> one in package.el format. Adding support package.el format in core means
>> that package developers only have to support one of these.
>
> What do you mean by "system for building packages"?  AFAIK, we don't
> build packages in core, we just byte-compile them and extract
> autoloads from them.

Yep, that's building them in the lisp sense -- also the documentation.
package.el does the same thing (generates autoloads, byte-compiles, and
additionally generates a -pkg.el file).

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:

>> I am suggesting that we enable the use of package.el as part of the
>> Emacs build to compile and generate the autoloads for packages in Emacs
>> core.
>
> Sounds good (IIUC).  So you are talking not about users having to use
> package.el but just the Emacs build using package.el.


Just so.



>> > 2. All Emacs Lisp code, apart from 3rd-party libraries, will continue
>> >    to be available in .../lisp/, so users can easily grep for it and
>> >    use `M-x find-library', without messing with packages.
>> 
>> This, I am suggesting we change, so that we have two locations for lisp
>> packages dependent on their file organisation. The lisp code will still
>> be available, though.
>
> I am no doubt a minority of one, but I do not want to have to use
> package.el at all, especially just to be able to use vanilla Emacs
> (standard distribution).


I'd agree with this.

As an aside, I'd like to add something that suggests packages you might
want to try based on what you do. So, if you open an ada file, Emacs
might suggest you install ada-mode from ELPA. But that is for the future.

>
>> > 3. The source-code directory structure under .../lisp/ does not become
>> >    any, or much, deeper.  So far, *.el, */*.el, and */*/*.el are
>> >    sufficient.
>> 
>> I would not want to change this.
>
> How does your answer here fit with your answer for #2?  I would like
> to find everything under .../lisp/ still, with a simple subdir structure.


Ah, that's a different thing. I am not suggesting we touch ./lisp at
all. So ./lisp would remain the same.

> But again, I'm no doubt a minority of one.  Just saying what I prefer.

So far, it looks like I am in a minority of one, rather than you.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> Sure, but it cannot change the fact that ELPA and Emacs core have
PL> different organisations because Emacs has a unified (or monolithic
PL> depending on your POV) directory structure, while ELPA has a per-package
PL> structure.

I understand that ELPA has per-package structure, but there's no requirement
for this to be a physical mapping.  As long as we know which files within
EMACS corresponding to the package files within the EPLA package, this is
sufficient.

So, if the mapping of EPLA-tree to Emacs-tree is not obvious by convention, we
could introduce a file to provide that mapping. It's doesn't need to be
either-or, simply because of current physical mappings.

    ELPA/foo/foo.el
    ELPA/foo/doc/foo.doc

could virtually map to:

    EMACS/lisp/foo.el
    EMACS/doc/foo.doc

by convention. Or more complex scenarios by a mapping file. Since we have a
full programming language at our behest, there is no limit to our
organizational capabilities. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>
> At the moment, no, there isn't (at least not until the -pkg.el file is
> built).
>
>> Or use a file-local variable as Eli proposed? So I think we get it
>> nearly for "free". Or what do you mean by "keeping track of"?
>
> Just what you think, yes. We need to distinguish between the two
> formats. Putting them different directories, with different make files
> is a trivial way to achieve this.

FWIW, if there is a strict naming and folder mapping convention, just by
looking at the list of ELPA-in-core package _names_, one can derive
which files belong to which package.  So an explicit "tagging" might not
even be required at all.

But I do not really understand when exactly we have to distinguish
between the two formats.  Isn't it more like a one way extraction of
ELPA package into core and then the job is done and distinction doesn't
really matter anymore?

> Just for what I say. Having the two package systems in different
> top-level (or lower-lever) directories makes life easier. Copying files
> from package.el format locations in core format would be possible but,
> again, complex.

Yes, there is an extra move files/directories step involved.  But I
think it finally makes the _user's_ life easier ;)

>> Whether it's confusing or not -- well you will find arguments against
>> all proposed approaches.  I think d) is easier because it doesn't divide
>> elisp source code based on where the sources actually come from.
>
> I understand that. But, unless we do something complex tests,
> documentation, icons, subsidiary files and so forth will be in different
> places for one style of packages than for the other.

Don't know if that really matters.  Well it would if we wouldn't put
resource files such as icons, subsidiary files and so forth at the same
location as the *.el files.  As packages may use 'load-file-name' to
locate these files.  FWIW, I would keep them next to the *.el files.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Alain Schneble <a.s@realize.ch> writes:
>>
>> At the moment, no, there isn't (at least not until the -pkg.el file is
>> built).
>>
>>> Or use a file-local variable as Eli proposed? So I think we get it
>>> nearly for "free". Or what do you mean by "keeping track of"?
>>
>> Just what you think, yes. We need to distinguish between the two
>> formats. Putting them different directories, with different make files
>> is a trivial way to achieve this.
>
> FWIW, if there is a strict naming and folder mapping convention, just by
> looking at the list of ELPA-in-core package _names_, one can derive
> which files belong to which package.  So an explicit "tagging" might not
> even be required at all.
>
> But I do not really understand when exactly we have to distinguish
> between the two formats.  Isn't it more like a one way extraction of
> ELPA package into core and then the job is done and distinction doesn't
> really matter anymore?

Not if we are using package.el to make the packages available. It is
package.el which sets the load path, loads the autoloads file, that sort
of thing.


>> Just for what I say. Having the two package systems in different
>> top-level (or lower-lever) directories makes life easier. Copying files
>> from package.el format locations in core format would be possible but,
>> again, complex.
>
> Yes, there is an extra move files/directories step involved.  But I
> think it finally makes the _user's_ life easier ;)

"user" here means "developer" - it doesn't make any difference for the
end user. For the developer, yes, it's probably easier for those who are
used to the lisp in the core, but harder for those who are used to
package.el format.



>> I understand that. But, unless we do something complex tests,
>> documentation, icons, subsidiary files and so forth will be in different
>> places for one style of packages than for the other.
>
> Don't know if that really matters.  Well it would if we wouldn't put
> resource files such as icons, subsidiary files and so forth at the same
> location as the *.el files.  As packages may use 'load-file-name' to
> locate these files.  FWIW, I would keep them next to the *.el files.

So would I, but that is not the directory layout for core. It is for package.el.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> PL> Sure, but it cannot change the fact that ELPA and Emacs core have
> PL> different organisations because Emacs has a unified (or monolithic
> PL> depending on your POV) directory structure, while ELPA has a per-package
> PL> structure.
>
> I understand that ELPA has per-package structure, but there's no requirement
> for this to be a physical mapping.  As long as we know which files within
> EMACS corresponding to the package files within the EPLA package, this is
> sufficient.
>
> So, if the mapping of EPLA-tree to Emacs-tree is not obvious by convention, we
> could introduce a file to provide that mapping. It's doesn't need to be
> either-or, simply because of current physical mappings.
>
>     ELPA/foo/foo.el
>     ELPA/foo/doc/foo.doc
>
> could virtually map to:
>
>     EMACS/lisp/foo.el
>     EMACS/doc/foo.doc
>
> by convention. Or more complex scenarios by a mapping file. Since we have a
> full programming language at our behest, there is no limit to our
> organizational capabilities. :)

Yes, sure. But as Emacs is already capable of building and loading
packages in package.el format, why not use that?

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> Yes, sure. But as Emacs is already capable of building and loading
PL> packages in package.el format, why not use that?

Because it comes with conventions and restrictions on placement that we may
not want to adopt now. I think the package.el discussion should be secondary
to how we include ELPA within Emacs core and the tarball.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>,  <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
> Date: Mon, 03 Oct 2016 13:32:34 +0100
> 
> > But I do not really understand when exactly we have to distinguish
> > between the two formats.  Isn't it more like a one way extraction of
> > ELPA package into core and then the job is done and distinction doesn't
> > really matter anymore?
> 
> Not if we are using package.el to make the packages available. It is
> package.el which sets the load path, loads the autoloads file, that sort
> of thing.

If it's package.el dictating this, then let's change it or extend it
to support what we need.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

>>> Just for what I say. Having the two package systems in different
>>> top-level (or lower-lever) directories makes life easier. Copying files
>>> from package.el format locations in core format would be possible but,
>>> again, complex.
>>
>> Yes, there is an extra move files/directories step involved.  But I
>> think it finally makes the _user's_ life easier ;)
>
> "user" here means "developer" - it doesn't make any difference for the
> end user. For the developer, yes, it's probably easier for those who are
> used to the lisp in the core, but harder for those who are used to
> package.el format.

It was not clear from what I wrote, but I was referring to a user --
whether developer or not -- using a release version of Emacs and that
potentially doesn't even want to use package.el to install additional
packages.  If we stick to the Emacs directory/file layout, it is a
unified layout that gets presented to her.  Where distictions between
core libraries and ELPA core packages aren't visible at that level.
That would be worth trying to achieve, I think.  (And in fact is how the
directory/files organization looks like in the current release, IIUC.)

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>
>> phillip.lord@russet.org.uk (Phillip Lord) writes:
>>
>>> Just what you think, yes. We need to distinguish between the two
>>> formats. Putting them different directories, with different make files
>>> is a trivial way to achieve this.
>>
>> FWIW, if there is a strict naming and folder mapping convention, just by
>> looking at the list of ELPA-in-core package _names_, one can derive
>> which files belong to which package.  So an explicit "tagging" might not
>> even be required at all.
>>
>> But I do not really understand when exactly we have to distinguish
>> between the two formats.  Isn't it more like a one way extraction of
>> ELPA package into core and then the job is done and distinction doesn't
>> really matter anymore?
>
> Not if we are using package.el to make the packages available. It is
> package.el which sets the load path, loads the autoloads file, that sort
> of thing.

After all, what would we gain from using package.el to do this
bootstrapping for the ELPA core packages?  If I understand correctly,
finder.el does populate package--builtins already today, based on the
files and directories in ./lisp.  Just automatically fetching all ELPA
core packages from the corresponding git repository (or repositories?),
extracting and moving files to the proper Emacs directories wouldn't
require any (or much) additional logic on that level.  Do I miss
something here?

>>> I understand that. But, unless we do something complex tests,
>>> documentation, icons, subsidiary files and so forth will be in different
>>> places for one style of packages than for the other.
>>
>> Don't know if that really matters.  Well it would if we wouldn't put
>> resource files such as icons, subsidiary files and so forth at the same
>> location as the *.el files.  As packages may use 'load-file-name' to
>> locate these files.  FWIW, I would keep them next to the *.el files.
>
> So would I, but that is not the directory layout for core. It is for package.el.

I would still move tests into ./test/automated/, for example.  And now
if I think of it, it would probably make sense to move resource files
(static data such as icons, schemas etc.) into ./etc/ and not into
./lisp/?  Is that where such files of non-ELPA, built-in libraries are
put in Emacs today?

Alain

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> I was referring to a user --
> whether developer or not -- using a release version of Emacs and that
> potentially doesn't even want to use package.el to install additional
> packages.  If we stick to the Emacs directory/file layout, it is a
> unified layout that gets presented to her.  Where distictions between
> core libraries and ELPA core packages aren't visible at that level.
> That would be worth trying to achieve, I think.  (And in fact is how the
> directory/files organization looks like in the current release, IIUC.)

Just what I was saying.  I would like to be able to access
and  use the distributed source files the same as in the
past, without needing to use package.el.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>>>> Just for what I say. Having the two package systems in different
>>>> top-level (or lower-lever) directories makes life easier. Copying files
>>>> from package.el format locations in core format would be possible but,
>>>> again, complex.
>>>
>>> Yes, there is an extra move files/directories step involved.  But I
>>> think it finally makes the _user's_ life easier ;)
>>
>> "user" here means "developer" - it doesn't make any difference for the
>> end user. For the developer, yes, it's probably easier for those who are
>> used to the lisp in the core, but harder for those who are used to
>> package.el format.
>
> It was not clear from what I wrote, but I was referring to a user --
> whether developer or not -- using a release version of Emacs and that
> potentially doesn't even want to use package.el to install additional
> packages.

I was never talking about the user manually using package.el to install
additional packages.

I was talking about using package.el as part of the build process to
compile package, and then using package.el as part of startup to
initialize packages in core.


> If we stick to the Emacs directory/file layout, it is a unified layout
> that gets presented to her. Where distictions between core libraries
> and ELPA core packages aren't visible at that level. That would be
> worth trying to achieve, I think. (And in fact is how the
> directory/files organization looks like in the current release, IIUC.)


AFAICT the layout of files inside the Emacs directory is, or should be,
more or less irrelevant to the end user of Emacs. We do not need to
maintain the current directory structure, to maintain the user
experience.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:

>> I was referring to a user --
>> whether developer or not -- using a release version of Emacs and that
>> potentially doesn't even want to use package.el to install additional
>> packages.  If we stick to the Emacs directory/file layout, it is a
>> unified layout that gets presented to her.  Where distictions between
>> core libraries and ELPA core packages aren't visible at that level.
>> That would be worth trying to achieve, I think.  (And in fact is how the
>> directory/files organization looks like in the current release, IIUC.)
>
> Just what I was saying.  I would like to be able to access
> and  use the distributed source files the same as in the
> past, without needing to use package.el.

We are talking cross purposes, I think.

There are two means to "use package.el". One is "use some of the
functions in the package.el file", and the other is "have the user
interact either through the API or through the UI with package.el".

My patch would mean that, during initialization the former would be
happening -- that is, Emacs would be using some package.el functions.
But not that latter -- as a user you would not need to interact with it.

Actually, your emacs is already doing the former (initializing
packages), even if there are no packages by default.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:
>> Not if we are using package.el to make the packages available. It is
>> package.el which sets the load path, loads the autoloads file, that sort
>> of thing.
>
> After all, what would we gain from using package.el to do this
> bootstrapping for the ELPA core packages?


Packages in already in package.el format can be directly used within
Emacs. This requires no changes in the file layout, and means that
packages will only be built with a single system (i.e. both Emacs core
and ELPA will be build with package.el).

As a secondary benefit, this means I can build and test an ELPA checkout
directly as part of the Emacs build, which should be useful for finding
regressions.


> If I understand correctly, finder.el does populate package--builtins
> already today, based on the files and directories in ./lisp. Just
> automatically fetching all ELPA core packages from the corresponding
> git repository (or repositories?), extracting and moving files to the
> proper Emacs directories wouldn't require any (or much) additional
> logic on that level. Do I miss something here?

If it all works, no. But I see no benefit from doing this.

It also, of course, means that files from ELPA would now be duplicated
in core Emacs because they would have been copied. So, when developing
Emacs, there would be version controlled .el source files and
non-version controlled copied .el files in the same location. You would
have to remember to edit the former, but not the latter.


>> So would I, but that is not the directory layout for core. It is for package.el.
>
> I would still move tests into ./test/automated/, for example.  And now
> if I think of it, it would probably make sense to move resource files
> (static data such as icons, schemas etc.) into ./etc/ and not into
> ./lisp/?  Is that where such files of non-ELPA, built-in libraries are
> put in Emacs today?

Yes, resource files are in ./etc

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>,  <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
> Date: Fri, 07 Oct 2016 17:29:08 +0100
> 
> Alain Schneble <a.s@realize.ch> writes:
> >> Not if we are using package.el to make the packages available. It is
> >> package.el which sets the load path, loads the autoloads file, that sort
> >> of thing.
> >
> > After all, what would we gain from using package.el to do this
> > bootstrapping for the ELPA core packages?
> 
> Packages in already in package.el format can be directly used within
> Emacs.

What do you mean by "directly used"?  Directly as opposed to what?

> This requires no changes in the file layout, and means that
> packages will only be built with a single system (i.e. both Emacs core
> and ELPA will be build with package.el).

Why would the Emacs build require package.el to do anything at all?

And what kind of build do you have in mind here?  We have:

  . build out of Git repo
  . build of the release tarball as distributed from ftp.gnu.org
  . build of the release tarball after updating some packages from ELPA

> As a secondary benefit, this means I can build and test an ELPA checkout
> directly as part of the Emacs build, which should be useful for finding
> regressions.

ELPA packages should be logically part of Emacs, just in a different
Git repo.  So this goal should be supported, of course.  But I don't
understand why it would require using a separate directory tree for
ELPA packages.

> It also, of course, means that files from ELPA would now be duplicated
> in core Emacs because they would have been copied.

??? Copied from where to where?  And why?  I don't understand why they
would need to be copied anywhere, they just need to be downloaded
directly to where they belong in the Emacs directory structure.

> So, when developing Emacs, there would be version controlled .el
> source files and non-version controlled copied .el files in the same
> location.

We already have that; see charscript.el.  Why having some moe
unversioned *.el files would hurt or be any different?

> You would have to remember to edit the former, but not the latter.

??? Unversioned files can be edited to your heart's content, they will
just be overwritten on the next update.  We successfully deal with
this with the generated files, I see no reasons why we couldn't do the
same with ELPA packages.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>>> Not if we are using package.el to make the packages available. It is
>>> package.el which sets the load path, loads the autoloads file, that sort
>>> of thing.
>>
>> After all, what would we gain from using package.el to do this
>> bootstrapping for the ELPA core packages?
>
> Packages in already in package.el format can be directly used within
> Emacs. This requires no changes in the file layout, and means that
> packages will only be built with a single system (i.e. both Emacs core
> and ELPA will be build with package.el).

Core libraries not from ELPA (e.g. url) will still be compiled using
non-package.el build.  So after all, we still have the "core library
build" approach.  Again, my point is to use that approach also for ELPA
core packages.  I would rather stick to that instead of introducing a
hybrid approach in Emacs core just to support ELPA core packages.

> As a secondary benefit, this means I can build and test an ELPA checkout
> directly as part of the Emacs build, which should be useful for finding
> regressions.

Sticking to the core directory layout wouldn't imply that this is not
possible.  We could again copy the files before the tests are run or
provide support for a development-time structure that more adheres to
the package.el structure.

>> If I understand correctly, finder.el does populate package--builtins
>> already today, based on the files and directories in ./lisp. Just
>> automatically fetching all ELPA core packages from the corresponding
>> git repository (or repositories?), extracting and moving files to the
>> proper Emacs directories wouldn't require any (or much) additional
>> logic on that level. Do I miss something here?
>
> If it all works, no. But I see no benefit from doing this.

I see the benefit that the release tarball directory layout will be
cleaner.

> It also, of course, means that files from ELPA would now be duplicated
> in core Emacs because they would have been copied. So, when developing
> Emacs, there would be version controlled .el source files and
> non-version controlled copied .el files in the same location. You would
> have to remember to edit the former, but not the latter.

I'm primarily concerned about changing the release tarball layout.
Again, if copying files really is a problem we could support a
development-time layout that removes this duplication.

>>> So would I, but that is not the directory layout for core. It is for package.el.
>>
>> I would still move tests into ./test/automated/, for example.  And now
>> if I think of it, it would probably make sense to move resource files
>> (static data such as icons, schemas etc.) into ./etc/ and not into
>> ./lisp/?  Is that where such files of non-ELPA, built-in libraries are
>> put in Emacs today?
>
> Yes, resource files are in ./etc

Thanks.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> I was never talking about the user manually using package.el to install
> additional packages.

Thanks for clarifying. I think I got that already earlier in this
discussion.

> I was talking about using package.el as part of the build process to
> compile package, and then using package.el as part of startup to
> initialize packages in core.

Where the code used to drive the build process comes from is a separate
question and isn't really relevant for the concerns I have.  But the
final results (incl. directory layout) matters, IMHO.  And with the way
you propose to use package.el it matters because it brings in a new
directory layout just for the ELPA core packages.

>> If we stick to the Emacs directory/file layout, it is a unified layout
>> that gets presented to her. Where distictions between core libraries
>> and ELPA core packages aren't visible at that level. That would be
>> worth trying to achieve, I think. (And in fact is how the
>> directory/files organization looks like in the current release, IIUC.)
>
> AFAICT the layout of files inside the Emacs directory is, or should be,
> more or less irrelevant to the end user of Emacs. We do not need to
> maintain the current directory structure, to maintain the user
> experience.

I don't think it's irrelevant.  But that's me.  Having a unified
directory layout certainly is a good thing, I think.  And as long as not
all Emacs built-in libraries are ELPA core packages, I'm more towards
keeping the current structure than bringing in a new one side-by-side.
(And AFAIU, making all libraries ELPA core packages isn't even a
long-term goal...)

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

> And what kind of build do you have in mind here?  We have:
>
>   . build out of Git repo
>   . build of the release tarball as distributed from ftp.gnu.org
>   . build of the release tarball after updating some packages from ELPA

What is the difference between the last and the second last point?

>> As a secondary benefit, this means I can build and test an ELPA checkout
>> directly as part of the Emacs build, which should be useful for finding
>> regressions.
>
> ELPA packages should be logically part of Emacs, just in a different
> Git repo.  So this goal should be supported, of course.  But I don't
> understand why it would require using a separate directory tree for
> ELPA packages.
>
>> It also, of course, means that files from ELPA would now be duplicated
>> in core Emacs because they would have been copied.
>
> ??? Copied from where to where?  And why?  I don't understand why they
> would need to be copied anywhere, they just need to be downloaded
> directly to where they belong in the Emacs directory structure.
>
>> So, when developing Emacs, there would be version controlled .el
>> source files and non-version controlled copied .el files in the same
>> location.
>
> We already have that; see charscript.el.  Why having some moe
> unversioned *.el files would hurt or be any different?
>
>> You would have to remember to edit the former, but not the latter.
>
> ??? Unversioned files can be edited to your heart's content, they will
> just be overwritten on the next update.  We successfully deal with
> this with the generated files, I see no reasons why we couldn't do the
> same with ELPA packages.

AFAIU, Phillip is concerned about the development process of ELPA core
packages.  While developing such a package, one wants to edit the
git-controlled files directly and probably also load these files instead
of the (git-uncontrolled) ones in the proper Emacs core location, where
they would reside at least in a tarball release.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> CC: Phillip Lord <phillip.lord@russet.org.uk>, <monnier@iro.umontreal.ca>,
> 	<emacs-devel@gnu.org>
> Date: Sat, 8 Oct 2016 12:57:54 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And what kind of build do you have in mind here?  We have:
> >
> >   . build out of Git repo
> >   . build of the release tarball as distributed from ftp.gnu.org
> >   . build of the release tarball after updating some packages from ELPA
> 
> What is the difference between the last and the second last point?

I don't know; maybe none at all.  I just wanted to be exhaustive, and
the last item is different in that it updates the files in the tarball
with the ones on ELPA.

> AFAIU, Phillip is concerned about the development process of ELPA core
> packages.  While developing such a package, one wants to edit the
> git-controlled files directly and probably also load these files instead
> of the (git-uncontrolled) ones in the proper Emacs core location, where
> they would reside at least in a tarball release.

The Right Way of doing this is exactly as we do with the likes of
charscript.el: edit the files that are their sources, in this case the
file in the ELPA repo, then commit the changes there, and finally
re-download them into the Emacs tree.  We are doing that all the time
with generated files.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alain Schneble <a.s@realize.ch>
>> CC: Phillip Lord <phillip.lord@russet.org.uk>, <monnier@iro.umontreal.ca>,
>> 	<emacs-devel@gnu.org>
>> Date: Sat, 8 Oct 2016 12:57:54 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > And what kind of build do you have in mind here?  We have:
>> >
>> >   . build out of Git repo
>> >   . build of the release tarball as distributed from ftp.gnu.org
>> >   . build of the release tarball after updating some packages from ELPA
>> 
>> What is the difference between the last and the second last point?
>
> I don't know; maybe none at all.  I just wanted to be exhaustive, and
> the last item is different in that it updates the files in the tarball
> with the ones on ELPA.

Aha, thanks for clarifying.

>> AFAIU, Phillip is concerned about the development process of ELPA core
>> packages.  While developing such a package, one wants to edit the
>> git-controlled files directly and probably also load these files instead
>> of the (git-uncontrolled) ones in the proper Emacs core location, where
>> they would reside at least in a tarball release.
>
> The Right Way of doing this is exactly as we do with the likes of
> charscript.el: edit the files that are their sources, in this case the
> file in the ELPA repo, then commit the changes there, and finally
> re-download them into the Emacs tree.  We are doing that all the time
> with generated files.

Do you mean committing the files with or without pushing to the remote
git repo?  I guess you mean including a push?  That would work of
course, but might be a bit cumbersome.  I guess somentimes one wants to
try it out prior to pushing to the remote.  (Of course, changes can
always be eval'ed in the running Emacs, regardless in which file they
are...).  Also, there must be a way to properly deal with branches.  Is
that supported by the workflow used with charscript.el?

Or if you meant without pushing at all, then none of the above would be
an issue, I think.  But the logic of getting ELPA core packages into the
Emacs directory tree would be a bit different, as it would have to get
the files from the local git clone.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> CC: <emacs-devel@gnu.org>, <monnier@iro.umontreal.ca>,
> 	<phillip.lord@russet.org.uk>
> Date: Sat, 8 Oct 2016 13:41:22 +0200
> 
> >> AFAIU, Phillip is concerned about the development process of ELPA core
> >> packages.  While developing such a package, one wants to edit the
> >> git-controlled files directly and probably also load these files instead
> >> of the (git-uncontrolled) ones in the proper Emacs core location, where
> >> they would reside at least in a tarball release.
> >
> > The Right Way of doing this is exactly as we do with the likes of
> > charscript.el: edit the files that are their sources, in this case the
> > file in the ELPA repo, then commit the changes there, and finally
> > re-download them into the Emacs tree.  We are doing that all the time
> > with generated files.
> 
> Do you mean committing the files with or without pushing to the remote
> git repo?  I guess you mean including a push?

Eventually, a push will be needed to get the change to everyone else,
of course.

> That would work of course, but might be a bit cumbersome.  I guess
> somentimes one wants to try it out prior to pushing to the remote.

I don't see anything that would preclude that in this case.  Try out
locally first, then commit and push later.  As usual.

> Or if you meant without pushing at all, then none of the above would be
> an issue, I think.  But the logic of getting ELPA core packages into the
> Emacs directory tree would be a bit different, as it would have to get
> the files from the local git clone.

I was hoping the modified package.el will help us get the files from
the repository, either local or remote, directly into the Emacs source
tree.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

> I was hoping the modified package.el will help us get the files from
> the repository, either local or remote, directly into the Emacs source
> tree.

You are right, supporting both cases would indeed make sense.  And
depending on how this is going to be implemented, these may not be much
different.  Especially if we would always clone all ELPA core package
repositories locally and then dispatch the files from there into the
Emacs source tree.  Or did you envision to just download the latest
(tagged?) revision and only have the repos cloned on demand?

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> CC: <phillip.lord@russet.org.uk>, <monnier@iro.umontreal.ca>,
> 	<emacs-devel@gnu.org>
> Date: Sat, 8 Oct 2016 15:33:06 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I was hoping the modified package.el will help us get the files from
> > the repository, either local or remote, directly into the Emacs source
> > tree.
> 
> You are right, supporting both cases would indeed make sense.  And
> depending on how this is going to be implemented, these may not be much
> different.  Especially if we would always clone all ELPA core package
> repositories locally and then dispatch the files from there into the
> Emacs source tree.  Or did you envision to just download the latest
> (tagged?) revision and only have the repos cloned on demand?

I didn't think about details this deep, only about general principles.

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> >> I was referring to a user --
> >> whether developer or not -- using a release version of Emacs and that
> >> potentially doesn't even want to use package.el to install additional
> >> packages.  If we stick to the Emacs directory/file layout, it is a
> >> unified layout that gets presented to her.  Where distictions between
> >> core libraries and ELPA core packages aren't visible at that level.
> >> That would be worth trying to achieve, I think.  (And in fact is how the
> >> directory/files organization looks like in the current release, IIUC.)
> >
> > Just what I was saying.  I would like to be able to access
> > and  use the distributed source files the same as in the
> > past, without needing to use package.el.
> 
> We are talking cross purposes, I think.

Not that I can see.

> There are two means to "use package.el". One is "use some of the
> functions in the package.el file", and the other is "have the user
> interact either through the API or through the UI with package.el".

It should be clear that I am talking about the latter - I do
not want to _have_ to do that.

> My patch would mean that, during initialization the former would be
> happening -- that is, Emacs would be using some package.el functions.
> But not that latter -- as a user you would not need to interact with it.
> 
> Actually, your emacs is already doing the former (initializing
> packages), even if there are no packages by default.

Again: Just what I was saying.

I too am talking about what _users_ do, need to do, and
want to do.

I do not object (in principle - the devil might be in the
resulting implementation) to Emacs using package.el under
the covers.  But see next...

I would also like to find the Lisp sources distributed with
Emacs (meaning at least those that have been distributed
in the past and are still part of Emacs) still all under
.../lisp/.

If "under the covers" means only that I do not need to
invoke package.el functions directly, but it also means that
I cannot find (grep) all distributed Emacs Lisp files living
peacefully under .../lisp/, that does not leave me a happy
camper, a priori.

I've been clear, I think, that I do not want to _need_ to
use package.el.  Myself.  As a user.  AND I would like to
just grep .../lisp/ and its subdirs, to find Emacs Lisp code.
That's all.  To me, the latter is part of not needing to
use package.el.

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> AFAICT the layout of files inside the Emacs directory is, or should be,
> more or less irrelevant to the end user of Emacs. We do not need to
> maintain the current directory structure, to maintain the user
> experience.

Why do you think so?  A somewhat large part of this user's user
experience involves interacting with that directory structure.

I would not consider it a benign change to mess with that structure.
Sometimes change that I am not in favor of need to be made, or
at least some think that they need to be made.  So far, I don't see
why a change in where Emacs Lisp files are stored (distributed) is
needed.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Colin Baxter]

On Sat, Oct 08 2016, Drew Adams wrote:

>> AFAICT the layout of files inside the Emacs directory is, or should be,
>> more or less irrelevant to the end user of Emacs. We do not need to
>> maintain the current directory structure, to maintain the user
>> experience.
>
> Why do you think so?  A somewhat large part of this user's user
> experience involves interacting with that directory structure.
>
> I would not consider it a benign change to mess with that structure.
> Sometimes change that I am not in favor of need to be made, or
> at least some think that they need to be made.  So far, I don't see
> why a change in where Emacs Lisp files are stored (distributed) is
> needed.

As a mere Emacs user, I totally agree. A feature of the present directory
structure I most appreciate is the fact that I can (and do) install
simultaneously several versions of Emacs. It always amazes me that I can
use them concurrently without any issues. I wouldn't want to lose that
functionality. My two-cent's worth.  

Best wishes,

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> AFAICT the layout of files inside the Emacs directory is, or should be,
PL> more or less irrelevant to the end user of Emacs. We do not need to
PL> maintain the current directory structure, to maintain the user experience.

I think we're getting hung up on package.el, rather than focusing on how to
best integrate ELPA with our development process. Many people, myself
included, don't want to see files start moving around just to suit
package.el's conventions. We can *always* map ELPA project files to custom
locations as needed, or even change package.el.

So let's get back to how we can adapt our build process to include ELPA (1) in
the release tarball (for "ELPA tarball" packages), and (2) in our mainline
development (for "ELPA core" packages).

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:
>> I was talking about using package.el as part of the build process to
>> compile package, and then using package.el as part of startup to
>> initialize packages in core.
>
> Where the code used to drive the build process comes from is a separate
> question and isn't really relevant for the concerns I have.  But the
> final results (incl. directory layout) matters, IMHO.  And with the way
> you propose to use package.el it matters because it brings in a new
> directory layout just for the ELPA core packages.

Yes, I would agree, although without the "just". I'm hoping that ELPA
becomes a more critical part of Emacs.


>>> If we stick to the Emacs directory/file layout, it is a unified layout
>>> that gets presented to her. Where distictions between core libraries
>>> and ELPA core packages aren't visible at that level. That would be
>>> worth trying to achieve, I think. (And in fact is how the
>>> directory/files organization looks like in the current release, IIUC.)
>>
>> AFAICT the layout of files inside the Emacs directory is, or should be,
>> more or less irrelevant to the end user of Emacs. We do not need to
>> maintain the current directory structure, to maintain the user
>> experience.
>
> I don't think it's irrelevant.  But that's me.  Having a unified
> directory layout certainly is a good thing, I think.  And as long as not
> all Emacs built-in libraries are ELPA core packages, I'm more towards
> keeping the current structure than bringing in a new one side-by-side.
> (And AFAIU, making all libraries ELPA core packages isn't even a
> long-term goal...)

"All libraries" isn't possible unless the bootstrap process is
re-written, but aside from that, I would think that having all packages
in package.el format would be a good thing, but that moving to it may
not be worth it.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:

>> AFAICT the layout of files inside the Emacs directory is, or should be,
>> more or less irrelevant to the end user of Emacs. We do not need to
>> maintain the current directory structure, to maintain the user
>> experience.
>
> Why do you think so?  A somewhat large part of this user's user
> experience involves interacting with that directory structure.
>
> I would not consider it a benign change to mess with that structure.

That's why I would not suggest moving existing files en masse.

> Sometimes change that I am not in favor of need to be made, or
> at least some think that they need to be made.  So far, I don't see
> why a change in where Emacs Lisp files are stored (distributed) is
> needed.

I think I have explained my reasoning here.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Colin Baxter <m43cap@yandex.com> writes:

> On Sat, Oct 08 2016, Drew Adams wrote:
>
>>> AFAICT the layout of files inside the Emacs directory is, or should be,
>>> more or less irrelevant to the end user of Emacs. We do not need to
>>> maintain the current directory structure, to maintain the user
>>> experience.
>>
>> Why do you think so?  A somewhat large part of this user's user
>> experience involves interacting with that directory structure.
>>
>> I would not consider it a benign change to mess with that structure.
>> Sometimes change that I am not in favor of need to be made, or
>> at least some think that they need to be made.  So far, I don't see
>> why a change in where Emacs Lisp files are stored (distributed) is
>> needed.
>
> As a mere Emacs user, I totally agree. A feature of the present directory
> structure I most appreciate is the fact that I can (and do) install
> simultaneously several versions of Emacs. It always amazes me that I can
> use them concurrently without any issues. I wouldn't want to lose that
> functionality. My two-cent's worth.  

This stems from the installation structure -- Emacs files are stored
under a versioned directory. Nothing I am suggesting would alter this.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> PL> AFAICT the layout of files inside the Emacs directory is, or should be,
> PL> more or less irrelevant to the end user of Emacs. We do not need to
> PL> maintain the current directory structure, to maintain the user experience.
>
> I think we're getting hung up on package.el, rather than focusing on how to
> best integrate ELPA with our development process. Many people, myself
> included, don't want to see files start moving around just to suit
> package.el's conventions. We can *always* map ELPA project files to custom
> locations as needed, or even change package.el.

Yes, this is true. I am suggesting a very simple mapping -- we use the
same structure as ELPA.


> So let's get back to how we can adapt our build process to include ELPA (1) in
> the release tarball (for "ELPA tarball" packages), and (2) in our mainline
> development (for "ELPA core" packages).

Still confused about this, so let me check that I have this correct.

An ELPA tarball package would be included in the tarball.

An ELPA core package would be including in the tarball, and other
packages would be free to use it as a dependency?

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:
> I would also like to find the Lisp sources distributed with
> Emacs (meaning at least those that have been distributed
> in the past and are still part of Emacs) still all under
> .../lisp/.
>
> If "under the covers" means only that I do not need to
> invoke package.el functions directly, but it also means that
> I cannot find (grep) all distributed Emacs Lisp files living
> peacefully under .../lisp/, that does not leave me a happy
> camper, a priori.
>
> I've been clear, I think, that I do not want to _need_ to
> use package.el.  Myself.  As a user.  AND I would like to
> just grep .../lisp/ and its subdirs, to find Emacs Lisp code.
> That's all.  To me, the latter is part of not needing to
> use package.el.


Yes, I agree, that greping ./lisp is convienient. My current proposed
directory structure would change this -- you'd have to grep the
top-level directory which includes lots of other stuff. From a normal
user perspective with an installed Emacs the directory above lisp
includes etc.

I will investigate moving my "package" directory into "./lisp/". It will
complicate the build somewhat, but it can be done.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Packages in already in package.el format can be directly used within
>> Emacs.
>
> What do you mean by "directly used"?  Directly as opposed to what?

Without installation -- that is, as part of core emacs.


>> This requires no changes in the file layout, and means that
>> packages will only be built with a single system (i.e. both Emacs core
>> and ELPA will be build with package.el).
>
> Why would the Emacs build require package.el to do anything at all?

Why not? It's a convienient way of building packages.

> And what kind of build do you have in mind here?  We have:
>
>   . build out of Git repo
>   . build of the release tarball as distributed from ftp.gnu.org
>   . build of the release tarball after updating some packages from ELPA

All of these, I think.



>> As a secondary benefit, this means I can build and test an ELPA checkout
>> directly as part of the Emacs build, which should be useful for finding
>> regressions.
>
> ELPA packages should be logically part of Emacs, just in a different
> Git repo.  So this goal should be supported, of course.  But I don't
> understand why it would require using a separate directory tree for
> ELPA packages.


It enables the convienience of taking a directory containing a package
from ELPA and putting it within the Emacs build unchanged. This seems
clean and simple. It should also enable "git submodule/subtree" type
options.


>> It also, of course, means that files from ELPA would now be duplicated
>> in core Emacs because they would have been copied.
>
> ??? Copied from where to where?  And why?  I don't understand why they
> would need to be copied anywhere, they just need to be downloaded
> directly to where they belong in the Emacs directory structure.

Downloading is copying right?

>> So, when developing Emacs, there would be version controlled .el
>> source files and non-version controlled copied .el files in the same
>> location.
>
> We already have that; see charscript.el.  Why having some moe
> unversioned *.el files would hurt or be any different?

There are generated .el files in Emacs, and there are specific cases
where this is necessary, but it's not a great thing. Mostly, I think,
.el files should be source code. At the moment, it's only a few files.
Making this more common seems bad.

>> You would have to remember to edit the former, but not the latter.
>
> ??? Unversioned files can be edited to your heart's content, they will
> just be overwritten on the next update.

Yep, which is bad, if you didn't intend this to happen.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Or if you meant without pushing at all, then none of the above would be
>> an issue, I think.  But the logic of getting ELPA core packages into the
>> Emacs directory tree would be a bit different, as it would have to get
>> the files from the local git clone.
>
> I was hoping the modified package.el will help us get the files from
> the repository, either local or remote, directly into the Emacs source
> tree.

Yes, I do not know how this would be be achieved at the moment. My hope
is that we can do some git cleverness so that checking out git emacs
would also pull the relevant bits of ELPA, and that when editing the
relevant bits of ELPA in the Emacs source, it should be possible to
commit directly back to ELPA.

Phil

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> > Sometimes change[s] that I am not in favor of need to be made, or
> > at least some think that they need to be made.  So far, I don't see
> > why a change in where Emacs Lisp files are stored (distributed) is
> > needed.
> 
> I think I have explained my reasoning here.

I must have missed it.  What is the reason?  Why is this needed?

If a proposed automatic use of package.el is _transparent_ to
users then that should include meaning that the Lisp libraries
it puts in place are put in place under .../lisp/, just as now.

It could be OK to add another subdir of .../lisp/ for some new
stuff that we really do not think logically belongs either
directly in .../lisp/ itself or under any of its existing subdirs.

But such a case would likely be (and should be) exceptional.
Usually, when a new library is added, it fits in one of the
existing subdirs well enough.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1412 bytes --]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

>> So let's get back to how we can adapt our build process to include ELPA (1)
>> in the release tarball (for "ELPA tarball" packages), and (2) in our
>> mainline development (for "ELPA core" packages).

> Still confused about this, so let me check that I have this correct.

> An ELPA tarball package would be included in the tarball.

Yes. Somewhere (probably in a file in Emacs.git), we'd have a MANIFEST file to
declares which ELPA packages are special with respect to development (core
ELPA), and which to the distribution (tarball ELPA). It could also include
other details, such as which Git version we intend to use, where the files
should be mapped to, etc.

A next step would be to invent this file, and then adapt our build process to
act on its contents.

Once we can show that this does what we want, we can pick a trial package
(say, org-mode), and then move it out of Emacs.git, and demonstrate that all
the same thing still happen as they did before, even though the source now
originates from ELPA.git instead of Emacs.git.

> An ELPA core package would be including in the tarball, and other packages
> would be free to use it as a dependency?

Exactly.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: a.s@realize.ch,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 09 Oct 2016 21:38:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Packages in already in package.el format can be directly used within
> >> Emacs.
> >
> > What do you mean by "directly used"?  Directly as opposed to what?
> 
> Without installation -- that is, as part of core emacs.

What do you mean by "installation" in this case?  Copying each file to
its place, as opposed to just replicating the same directory
structure?  Or do you mean something else?

> >> This requires no changes in the file layout, and means that
> >> packages will only be built with a single system (i.e. both Emacs core
> >> and ELPA will be build with package.el).
> >
> > Why would the Emacs build require package.el to do anything at all?
> 
> Why not? It's a convienient way of building packages.

If that convenience comes at a price of making our directory structure
harder to work with, it's more like INconvenience.

> > And what kind of build do you have in mind here?  We have:
> >
> >   . build out of Git repo
> >   . build of the release tarball as distributed from ftp.gnu.org
> >   . build of the release tarball after updating some packages from ELPA
> 
> All of these, I think.

Each one could have a separate solution.  For example, the first one
could be done with Git commands, bypassing package.el entirely, or
with Git commands followed by something that package.el does to
populate the right directories.

> > ELPA packages should be logically part of Emacs, just in a different
> > Git repo.  So this goal should be supported, of course.  But I don't
> > understand why it would require using a separate directory tree for
> > ELPA packages.
> 
> It enables the convienience of taking a directory containing a package
> from ELPA and putting it within the Emacs build unchanged. This seems
> clean and simple. It should also enable "git submodule/subtree" type
> options.

See above: it could be easy to do, but be a nuisance in the long run.
So it is IMO incorrect to consider only the short-term convenience of
putting together the necessary machinery, we need to consider the
long-term convenience as well, if not first of all.

> >> It also, of course, means that files from ELPA would now be duplicated
> >> in core Emacs because they would have been copied.
> >
> > ??? Copied from where to where?  And why?  I don't understand why they
> > would need to be copied anywhere, they just need to be downloaded
> > directly to where they belong in the Emacs directory structure.
> 
> Downloading is copying right?

That price cannot be avoided, since ELPA is a different repository.
Right?

> >> So, when developing Emacs, there would be version controlled .el
> >> source files and non-version controlled copied .el files in the same
> >> location.
> >
> > We already have that; see charscript.el.  Why having some moe
> > unversioned *.el files would hurt or be any different?
> 
> There are generated .el files in Emacs, and there are specific cases
> where this is necessary, but it's not a great thing. Mostly, I think,
> .el files should be source code. At the moment, it's only a few files.
> Making this more common seems bad.

I disagree.  I see no reason to consider generated files "bad".  And
there's no fundamental difference between generating them by C or Lisp
programs and "generating" them by using Git commands.

> >> You would have to remember to edit the former, but not the latter.
> >
> > ??? Unversioned files can be edited to your heart's content, they will
> > just be overwritten on the next update.
> 
> Yep, which is bad, if you didn't intend this to happen.

We deal with this "badness" every day.  So I think this is a downside
that is very minor and can be disregarded when more serious issues are
at hand.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Alain Schneble <a.s@realize.ch>,  emacs-devel@gnu.org,  monnier@iro.umontreal.ca
> Date: Sun, 09 Oct 2016 21:42:59 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Or if you meant without pushing at all, then none of the above would be
> >> an issue, I think.  But the logic of getting ELPA core packages into the
> >> Emacs directory tree would be a bit different, as it would have to get
> >> the files from the local git clone.
> >
> > I was hoping the modified package.el will help us get the files from
> > the repository, either local or remote, directly into the Emacs source
> > tree.
> 
> Yes, I do not know how this would be be achieved at the moment.

One possibility is to have each ELPA package tell package.el where its
files should live in the Emacs tree.  For example, each package could
have a standard-named file with a Lisp data structure that specifies
this.  package.el would then use this information to place each file
where it belongs.  We could also have a set of reasonable defaults;
for example the tests should by default go to the directory under
test/lisp/ whose name is identical to the lisp/ subdirectory of the
Lisp files.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>> So let's get back to how we can adapt our build process to include ELPA (1)
>>> in the release tarball (for "ELPA tarball" packages), and (2) in our
>>> mainline development (for "ELPA core" packages).
>
>> Still confused about this, so let me check that I have this correct.
>
>> An ELPA tarball package would be included in the tarball.
>
> Yes. Somewhere (probably in a file in Emacs.git), we'd have a MANIFEST file to
> declares which ELPA packages are special with respect to development (core
> ELPA), and which to the distribution (tarball ELPA). It could also include
> other details, such as which Git version we intend to use, where the files
> should be mapped to, etc.


Just to be explicit, this means that we have to use git to grab the
relevant files from ELPA -- we cannot, for example, use package.el to
pull down files from the ELPA website, bcause it doesn't support
versions at the moment.

>> An ELPA core package would be including in the tarball, and other packages
>> would be free to use it as a dependency?
>
> Exactly.

All makes sense, but still a bit confused about how we would decide
whether a package would be tarball or core. What would the criteria be?

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Drew Adams <drew.adams@oracle.com> writes:

>> > Sometimes change[s] that I am not in favor of need to be made, or
>> > at least some think that they need to be made.  So far, I don't see
>> > why a change in where Emacs Lisp files are stored (distributed) is
>> > needed.
>> 
>> I think I have explained my reasoning here.
>
> I must have missed it.  What is the reason?  Why is this needed?
>
> If a proposed automatic use of package.el is _transparent_ to
> users then that should include meaning that the Lisp libraries
> it puts in place are put in place under .../lisp/, just as now.
>
> It could be OK to add another subdir of .../lisp/ for some new
> stuff that we really do not think logically belongs either
> directly in .../lisp/ itself or under any of its existing subdirs.
>
> But such a case would likely be (and should be) exceptional.
> Usually, when a new library is added, it fits in one of the
> existing subdirs well enough.

Simply because my proposed usage of package.el would use it to build the
packages, and these have a different structure and build from the files
in the lisp.

Anyway, your point about grepping is valid. I can and will move the
packages directory that I am proposing from top-level to lisp. If
grepping all lisp files is the issue, then this will solve that problem.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Alain Schneble <a.s@realize.ch>,  emacs-devel@gnu.org,  monnier@iro.umontreal.ca
>> Date: Sun, 09 Oct 2016 21:42:59 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> Or if you meant without pushing at all, then none of the above would be
>> >> an issue, I think.  But the logic of getting ELPA core packages into the
>> >> Emacs directory tree would be a bit different, as it would have to get
>> >> the files from the local git clone.
>> >
>> > I was hoping the modified package.el will help us get the files from
>> > the repository, either local or remote, directly into the Emacs source
>> > tree.
>> 
>> Yes, I do not know how this would be be achieved at the moment.
>
> One possibility is to have each ELPA package tell package.el where its
> files should live in the Emacs tree.  For example, each package could
> have a standard-named file with a Lisp data structure that specifies
> this.  package.el would then use this information to place each file
> where it belongs.

If we are moving files into the current directory structure, I'm not
sure we need package.el at all. It would be doing something totally
different from what it does at the moment (which is select, download,
install and initialize packages).


> We could also have a set of reasonable defaults; for example the tests
> should by default go to the directory under test/lisp/ whose name is
> identical to the lisp/ subdirectory of the Lisp files.

I think that this is essentially what ever we choose, although
personally, I would not have defaults, just a set of standards which
need to be followed.

We have some of this with ELPA already; I've already noted that there is
no standard location for test files in ELPA, and it is something I would
like to add.

Phil

RE: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Drew Adams]

> Anyway, your point about grepping is valid. I can and will move the
> packages directory that I am proposing from top-level to lisp. If
> grepping all lisp files is the issue, then this will solve that problem.

Great.  Thank you, for that and for the explanations.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

>> Yes. Somewhere (probably in a file in Emacs.git), we'd have a MANIFEST file
>> to declares which ELPA packages are special with respect to development
>> (core ELPA), and which to the distribution (tarball ELPA). It could also
>> include other details, such as which Git version we intend to use, where
>> the files should be mapped to, etc.

PL> Just to be explicit, this means that we have to use git to grab the
PL> relevant files from ELPA -- we cannot, for example, use package.el to pull
PL> down files from the ELPA website, bcause it doesn't support versions at
PL> the moment.

That's fine with me for the present time.

PL> All makes sense, but still a bit confused about how we would decide
PL> whether a package would be tarball or core. What would the criteria be?

Whatever the developers here decide. Someone will propose that a package be
included, and we'll agree or disagree on each point.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> If we are moving files into the current directory structure, I'm not sure
PL> we need package.el at all. It would be doing something totally different
PL> from what it does at the moment (which is select, download, install and
PL> initialize packages).

Then let's not involve package.el at the beginning of this effort. It's
apparently creating much more confusion than is necessary.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
> PL> All makes sense, but still a bit confused about how we would decide
> PL> whether a package would be tarball or core. What would the criteria be?
>
> Whatever the developers here decide. Someone will propose that a package be
> included, and we'll agree or disagree on each point.


That doesn't really answer the question:-)

I'm wondering on what basis we would make the distinction between core
and tarball. Potential usefulness, longevity, quality etc?

I'm not asking because I expect to have completely clear criteria at
this stage, but because I want to know that there are some criteria. If
there are not, then we may be making an unnecessary distinction.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> PL> If we are moving files into the current directory structure, I'm not sure
> PL> we need package.el at all. It would be doing something totally different
> PL> from what it does at the moment (which is select, download, install and
> PL> initialize packages).
>
> Then let's not involve package.el at the beginning of this effort. It's
> apparently creating much more confusion than is necessary.


I would not agree with this; at a code level, it's made things very
simple. There is a one-to-one mapping between files in ELPA and where
they reside in the Emacs build, and there are several easy ways to get
the files from ELPA into the build, including pre-existing git based
tools which may make development very convienient. As a nice side effect
it also lets me build and test all packages in an ELPA checkout from
inside an Emacs build, which would be a useful thing to be able to do.

So far, the main objection has been to the use of a new top-level
directory (which I've can and will move -- it's not necessary to have
top-level, although I still think its nicer).

My own feeling is that for an emacs-devel discussion, it was quite low
on the confusion stakes.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> I'm wondering on what basis we would make the distinction between core and
PL> tarball. Potential usefulness, longevity, quality etc?

This point has yet to be determined. I think that for now it will remain a
maintainer decision, based on feedback from the developers here. I don't think
it's useful to define a "general principle" to make the decision for us, yet.
There aren't that many packages signed over to the FSF, after all.

PL> I'm not asking because I expect to have completely clear criteria at this
PL> stage, but because I want to know that there are some criteria. If there
PL> are not, then we may be making an unnecessary distinction.

Ok, here's my thinking on what should move to ELPA tarball:

  1. Package is self-contained (that is, no packages in Emacs depend on it).
  2. Package is currently under development by an outside team.
  3. Package regularly presents us with version updates.

Org-mode fits this description to a T, and so it would move to tarball ELPA.
This should in no way be seen as a "downgrading" of the importance or role of
Org-mode within Emacs. It is solely to facilitate easier integrate of new
changes from the Org-mode team, and to relieve Emacs.git from having to track
changes in its files.

Gnus and CEDET are other prime candidates for tarball ELPA.

Calc and Eshell almost fit this description, except they are both very stable
now, so there's not much to be gained from moving them.

stream.el is something that I think would be a candidate for "core ELPA": It
fits the above criteria, but adds a fourth:

  4. We'd like for other core Emacs packages to be able to use it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> So far, the main objection has been to the use of a new top-level
PL> directory (which I've can and will move -- it's not necessary to have
PL> top-level, although I still think its nicer).

OK, if we can resolve this point, then, I'm happy to consider its use from the
beginning.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

PL> I'm wondering on what basis we would make the distinction between core and
PL> tarball.  Potential usefulness, longevity, quality etc?
> This point has yet to be determined. I think that for now it will remain a
> maintainer decision, based on feedback from the developers here. I don't think
> it's useful to define a "general principle" to make the decision for us, yet.
> There aren't that many packages signed over to the FSF, after all.

[...]

> stream.el is something that I think would be a candidate for "core ELPA": It
> fits the above criteria, but adds a fourth:
>
>   4. We'd like for other core Emacs packages to be able to use it.

I think this answers Phillip's question: the main difference is whether
Emacs's packages can (and/or do) depend on it.

There are several gradations of dependency, of course:

- Can src/emacs be dumped without that package?
- Can all of lisp/**/*.elc be built without that package?
- Can all of lisp/**/*.el be used without that package?

If all three answers are yes, then it doesn't need to be "core".
If all three are no, then it definitely needs to be "core".
Between the two, well, ...


        Stefan

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> There are several gradations of dependency, of course:

SM> - Can src/emacs be dumped without that package?
SM> - Can all of lisp/**/*.elc be built without that package?
SM> - Can all of lisp/**/*.el be used without that package?

SM> If all three answers are yes, then it doesn't need to be "core". If all
SM> three are no, then it definitely needs to be "core". Between the two,
SM> well, ...

This is a great distinction, Stefan. I guess we have several degrees of
propinquity in core:

  0. It's in C
  1. Emacs Lisp that is dumped into src/emacs
  2. Emacs Lisp that is required to compile all of lisp/
  3. Emacs Lisp that is required to use all of lisp/
  4. Emacs Lisp that is in core, but stands alone
  5. Emacs Lisp that should be in the distribution tarball          ELPA tarball
  6. Emacs Lisp that should be installed using M-x package-install  ELPA
  7. Emacs Lisp that is not signed over to the FSF

The reason to move something from levels 1-3 into ELPA is managerial: to more
easily accept version updates from an outside development team. Something
stable that has become "part of Emacs" should still fully move into core.

The reason not to move #4 into ELPA is historical: It's a stable package and
we don't gain anything by doing so.

The reason to move #5 into ELPA is likewise managerial, but is much easier to
justify: the development and use of the package was largely separate from
Emacs anyway, except for added testing by being part of the test builds.

#6 stays in ELPA.

#7 contains a ton of code that we'd like to move into ELPA, and maybe it'll be
easier to encourage it, once people see the new attention we're paying here.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> So far, the main objection has been to the use of a new top-level
> directory (which I've can and will move -- it's not necessary to have
> top-level, although I still think its nicer).

I still do not really understand why we shall introduce such a new
layout, especially in the release Emacs tarballs published on
https://ftp.gnu.org/gnu/emacs/, and even more after installing them.

As an example, if I do ./configure --prefix=/usr/local followed by make
and make install today, I'll end up with a directory layout of org-mode
related files like this:

/usr/local/share/info/org.info.gz
/usr/local/share/emacs/26.0.50/etc/ORG-NEWS
/usr/local/share/emacs/26.0.50/etc/org/README
/usr/local/share/emacs/26.0.50/etc/org/.*\.xml
/usr/local/share/emacs/26.0.50/etc/refcards/orgcard.tex
/usr/local/share/emacs/26.0.50/lisp/org/.*\.(el\.gz|elc)

Unless I'm misundarstanding you, using your approach, all files will end
up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
configuration I gave above).

Do you really want to give up this standard file structure?

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Alain Schneble <a.s@realize.ch> writes:

> Unless I'm misundarstanding you, using your approach, all files will end
> up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I meant /usr/local/share/emacs/26.0.50/lisp/packages/org, of course.

> configuration I gave above).

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

>>>>> "AS" == Alain Schneble <a.s@realize.ch> writes:

AS> /usr/local/share/info/org.info.gz
AS> /usr/local/share/emacs/26.0.50/etc/ORG-NEWS
AS> /usr/local/share/emacs/26.0.50/etc/org/README
AS> /usr/local/share/emacs/26.0.50/etc/org/.*\.xml
AS> /usr/local/share/emacs/26.0.50/etc/refcards/orgcard.tex
AS> /usr/local/share/emacs/26.0.50/lisp/org/.*\.(el\.gz|elc)

AS> Unless I'm misundarstanding you, using your approach, all files will end
AS> up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
AS> configuration I gave above).

AS> Do you really want to give up this standard file structure?

I think we should *not* give up the standard file structure. At least not now.
Using ELPA should feel "first class" for those authors contributing packages
that are to become part of the standard distribution.

Perhaps in future, for the sake of ease-of-upgrading, we'd want to change this
policy, but one thing at a time.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Thu, 13 Oct 2016 15:29:47 -0700
> 
> I think we should *not* give up the standard file structure. At least not now.
> Using ELPA should feel "first class" for those authors contributing packages
> that are to become part of the standard distribution.

100% agreement.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> PL> I'm wondering on what basis we would make the distinction between core and
> PL> tarball.  Potential usefulness, longevity, quality etc?
>> This point has yet to be determined. I think that for now it will remain a
>> maintainer decision, based on feedback from the developers here. I don't think
>> it's useful to define a "general principle" to make the decision for us, yet.
>> There aren't that many packages signed over to the FSF, after all.
>
> [...]
>
>> stream.el is something that I think would be a candidate for "core ELPA": It
>> fits the above criteria, but adds a fourth:
>>
>>   4. We'd like for other core Emacs packages to be able to use it.
>
> I think this answers Phillip's question: the main difference is whether
> Emacs's packages can (and/or do) depend on it.
>
> There are several gradations of dependency, of course:
>
> - Can src/emacs be dumped without that package?
> - Can all of lisp/**/*.elc be built without that package?
> - Can all of lisp/**/*.el be used without that package?
>
> If all three answers are yes, then it doesn't need to be "core".
> If all three are no, then it definitely needs to be "core".
> Between the two, well, ...


I guess, the decision is more "whether we can reasonable expect other
packages to depend on it" -- stream.el, being new, is not going to
depended on by anything in core.

You'll note, though, that package.el format packages can, and do,
depend on each other. It's just that the dependency mechanism is
slightly different: package.el format packages need an additional line
in the header. Core packages just need a require (or nothing at all for
autoloads).

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> So far, the main objection has been to the use of a new top-level
>> directory (which I've can and will move -- it's not necessary to have
>> top-level, although I still think its nicer).
>
> I still do not really understand why we shall introduce such a new
> layout, especially in the release Emacs tarballs published on
> https://ftp.gnu.org/gnu/emacs/, and even more after installing them.
>
> As an example, if I do ./configure --prefix=/usr/local followed by make
> and make install today, I'll end up with a directory layout of org-mode
> related files like this:
>
> /usr/local/share/info/org.info.gz
> /usr/local/share/emacs/26.0.50/etc/ORG-NEWS
> /usr/local/share/emacs/26.0.50/etc/org/README
> /usr/local/share/emacs/26.0.50/etc/org/.*\.xml
> /usr/local/share/emacs/26.0.50/etc/refcards/orgcard.tex
> /usr/local/share/emacs/26.0.50/lisp/org/.*\.(el\.gz|elc)
>
> Unless I'm misundarstanding you, using your approach, all files will end
> up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
> configuration I gave above).
>
> Do you really want to give up this standard file structure?


Yes, because it is not standard. It's one of two standards.

If you install org with package.el, then you get

~/.emacs.d/elpa/org/org
~/.emacs.d/elpa/org/etc/ORG-NEWS
~/.emacs.d/elpa/org/org.el

So, org-mode has to support two independent directory layouts. If we use
package.el as part of the core or tarball emacs build, then org-mode has
to support only one directory layout.

The Emacs build will, for the foreseeable future, have to support two
layouts, that is true. But, Emacs already does and it is (or rather was)
relatively easy to add to the build.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "AS" == Alain Schneble <a.s@realize.ch> writes:
>
> AS> /usr/local/share/info/org.info.gz
> AS> /usr/local/share/emacs/26.0.50/etc/ORG-NEWS
> AS> /usr/local/share/emacs/26.0.50/etc/org/README
> AS> /usr/local/share/emacs/26.0.50/etc/org/.*\.xml
> AS> /usr/local/share/emacs/26.0.50/etc/refcards/orgcard.tex
> AS> /usr/local/share/emacs/26.0.50/lisp/org/.*\.(el\.gz|elc)
>
> AS> Unless I'm misundarstanding you, using your approach, all files will end
> AS> up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
> AS> configuration I gave above).
>
> AS> Do you really want to give up this standard file structure?
>
> I think we should *not* give up the standard file structure. At least not now.
> Using ELPA should feel "first class" for those authors contributing packages
> that are to become part of the standard distribution.

It can cut both ways, of course. It may well be that packages not in
ELPA feel second class, because they are not "real" packages.

I think the next step is to canvas opinion from other people. I know for
myself as a package developer which I would prefer, but that's me. As
we're discussing org a lot here, probably that would be a good place to
start.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Phillip Lord <phillip.lord@russet.org.uk>, Eli Zaretskii <eliz@gnu.org>,
>> monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Thu, 13 Oct 2016 15:29:47 -0700
>> 
>> I think we should *not* give up the standard file structure. At least not now.
>> Using ELPA should feel "first class" for those authors contributing packages
>> that are to become part of the standard distribution.
>
> 100% agreement.

Can you tell me why, though? Drew complained about grepability. What are
the other reasons?

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

>> Do you really want to give up this standard file structure?
>
> Yes, because it is not standard. It's one of two standards.
>
> If you install org with package.el, then you get
>
> ~/.emacs.d/elpa/org/org
> ~/.emacs.d/elpa/org/etc/ORG-NEWS
> ~/.emacs.d/elpa/org/org.el

I have nothing against this structure for user-added packages.  Because
it is up to the user to decide whether to use package.el or not.  If she
chooses to use it and install *additional* packages, then I think it is
legitimate to keep and use this ~/.emacs.d/elpa/ structure.

> So, org-mode has to support two independent directory layouts. If we use
> package.el as part of the core or tarball emacs build, then org-mode has
> to support only one directory layout.

Maybe I'm naive, but I don't see an issue in supporting both layouts.
The ~/.emacs.d/elpa/ layout, e.g. to support (old?) Emacs
tarball/distributions that do not include an ELPA tarball (or core)
package like org and which the user installs manually using package.el.
And the current layout for all ELPA core and ELPA tarball packages that
will be part of Emacs tarball in upcoming releases.

> The Emacs build will, for the foreseeable future, have to support two
> layouts, that is true. But, Emacs already does and it is (or rather was)
> relatively easy to add to the build.

Maybe or maybe not.  If we would use the current structure for ELPA core
and ELPA tarball packages, then the build wouldn't have to support the
/lisp/packages/[package] structure that follows the
~/.emacs.d/elpa/[package] layout.  It would only have to know how to
copy the package files to the proper locations.

Alain

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "AS" == Alain Schneble <a.s@realize.ch> writes:
>
> AS> /usr/local/share/info/org.info.gz
> AS> /usr/local/share/emacs/26.0.50/etc/ORG-NEWS
> AS> /usr/local/share/emacs/26.0.50/etc/org/README
> AS> /usr/local/share/emacs/26.0.50/etc/org/.*\.xml
> AS> /usr/local/share/emacs/26.0.50/etc/refcards/orgcard.tex
> AS> /usr/local/share/emacs/26.0.50/lisp/org/.*\.(el\.gz|elc)
>
> AS> Unless I'm misundarstanding you, using your approach, all files will end
> AS> up in /usr/local/share/emacs/26.0.50/lisp/org (using the example
> AS> configuration I gave above).
>
> AS> Do you really want to give up this standard file structure?
>
> I think we should *not* give up the standard file structure. At least not now.
> Using ELPA should feel "first class" for those authors contributing packages
> that are to become part of the standard distribution.
>
> Perhaps in future, for the sake of ease-of-upgrading, we'd want to change this
> policy, but one thing at a time.

It's worth considering what ease-of-upgrading means. At the moment, for
example, if I install org-mode with ELPA package.el generates all the
autoloads for org-mode and sticks them into a file. All is good.

But the old autoloads are still there, stuck in loaddefs.el in non-user
space, non-removable or changable. Functions that are in the
/usr/share/emacs installation but NOT in the ~/.emacs.d/elpa
installation will remain autoloaded, even though they should not be.

If org mode were it's own directory, with it's own org-autoloads.el,
then package.el could (and already does I believe) just not autoload any
of the functions from the old version of org-mode, nor put any of its
files into load-path.

It seems to me that one of the major use cases for adding core/tarball
ELPA support is enable packages to be present by default in an Emacs
tarball download, while still allowing upgrading from ELPA (or the
org-mode repo).

I cannot see how to address this using the "copy files from ELPA into
core" build, at least not with significant replumbing of the current
emacs build; perhaps someone else can, but it is worth considering this
when thinking about the valid but generic concerns I am hearing about
change.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>>> Do you really want to give up this standard file structure?
>>
>> Yes, because it is not standard. It's one of two standards.
>>
> If she chooses to use it and install *additional* packages, then I
> think it is legitimate to keep and use this ~/.emacs.d/elpa/
> structure.

Or upgrades existing ones.



>> So, org-mode has to support two independent directory layouts. If we use
>> package.el as part of the core or tarball emacs build, then org-mode has
>> to support only one directory layout.
>
> Maybe I'm naive, but I don't see an issue in supporting both layouts.

Two layouts have to be supported. It's just a question of who has to
support them. In my scheme, it's core emacs, in your scheme its
individual package developers.

Probably in most cases, it's not a big deal, and it will happen
automatically.

>> The Emacs build will, for the foreseeable future, have to support two
>> layouts, that is true. But, Emacs already does and it is (or rather was)
>> relatively easy to add to the build.
>
> Maybe or maybe not.  If we would use the current structure for ELPA core
> and ELPA tarball packages, then the build wouldn't have to support the
> /lisp/packages/[package] structure that follows the
> ~/.emacs.d/elpa/[package] layout.  It would only have to know how to
> copy the package files to the proper locations.


I think the semantics of understanding how to copy the package files to
the legacy (er, proper) locations is pretty much the same as package.el
already has just for loading them.

Oh, and, all of this copying has to happen *before* we generate
loaddefs.el which means it has be done with bootstrap-emacs, which is a
tricky bit of the build to be fiddling with.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>,  <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
> Date: Fri, 14 Oct 2016 09:20:14 +0100
> 
> > Do you really want to give up this standard file structure?
> 
> 
> Yes, because it is not standard. It's one of two standards.
> 
> If you install org with package.el, then you get
> 
> ~/.emacs.d/elpa/org/org
> ~/.emacs.d/elpa/org/etc/ORG-NEWS
> ~/.emacs.d/elpa/org/org.el
> 
> So, org-mode has to support two independent directory layouts.

But Org already supports those two formats.  We don't require Org to
do anything that it doesn't already do.  So staying with the current
structure of lisp/ in the Emacs tree doesn't add any new requirements.

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: John Wiegley <jwiegley@gmail.com>,  a.s@realize.ch,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 14 Oct 2016 09:25:13 +0100
> 
> >> I think we should *not* give up the standard file structure. At least not now.
> >> Using ELPA should feel "first class" for those authors contributing packages
> >> that are to become part of the standard distribution.
> >
> > 100% agreement.
> 
> Can you tell me why, though? Drew complained about grepability. What are
> the other reasons?

The structure of the Emacs lisp/ directory is well-thought and exists
for many years with only minor changes.  It has some underlying logic,
which allows one in most cases to know where a certain package lives.
This is important not just for grepping, but also for visiting the
files and any operation that requires its full file name.  Having some
files outside of this structure will make working with those files
more annoying.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Andy Moreton]

On Fri 14 Oct 2016, Eli Zaretskii wrote:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Eli Zaretskii <eliz@gnu.org>,  <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
>> Date: Fri, 14 Oct 2016 09:20:14 +0100
>> 
>> > Do you really want to give up this standard file structure?
>> 
>> 
>> Yes, because it is not standard. It's one of two standards.
>> 
>> If you install org with package.el, then you get
>> 
>> ~/.emacs.d/elpa/org/org
>> ~/.emacs.d/elpa/org/etc/ORG-NEWS
>> ~/.emacs.d/elpa/org/org.el
>> 
>> So, org-mode has to support two independent directory layouts.
>
> But Org already supports those two formats.  We don't require Org to
> do anything that it doesn't already do.  So staying with the current
> structure of lisp/ in the Emacs tree doesn't add any new requirements.

But that would do nothing to reduce the unnecessary and duplicated
packaging work.

Keeping each package in ELPA format ensures that replacing the package
can be done easily, as everything is isolated in a single directory. If
the package is shipped in the emacs tarball and the user then upgrades
to a newer version from ELPA, only the load path needs to change.

In additon, the user can easily compare the changes bewteen the package
version shipped in the emacs tarball and the updated one fetched from
ELPA, as the package layout is the same.

There are many more users of emacs than developers, so the design
should be aimed at utility and convenience for users.

    AndyM

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 14 Oct 2016 14:51:15 +0100
> 
> >> ~/.emacs.d/elpa/org/org
> >> ~/.emacs.d/elpa/org/etc/ORG-NEWS
> >> ~/.emacs.d/elpa/org/org.el
> >> 
> >> So, org-mode has to support two independent directory layouts.
> >
> > But Org already supports those two formats.  We don't require Org to
> > do anything that it doesn't already do.  So staying with the current
> > structure of lisp/ in the Emacs tree doesn't add any new requirements.
> 
> But that would do nothing to reduce the unnecessary and duplicated
> packaging work.

I'm not sure I understand what duplicated work you have in mind.  Org
developers will still put Org on ELPA in the same directory structure
they do now.  When Org is imported into Emacs, as part of building a
release tarball or as part of building from the repository, each file
will be put in its natural place in the Emacs source tree, either by
package.el or by some other means, and will replace any previous Org
files, if there were such.

When end-users want to upgrade to a version of Org newer than what
they have bundled in the latest Emacs release, the Org files will be
placed under ~/.emacs.d/elpa/org/, as they are now.

> Keeping each package in ELPA format ensures that replacing the package
> can be done easily, as everything is isolated in a single directory.

I see no particular difficulties in putting several files into several
directories, software can and does do that all the time.  It's not
like we will be asking users to do this manually.

> If the package is shipped in the emacs tarball and the user then
> upgrades to a newer version from ELPA, only the load path needs to
> change.

This part doesn't need to change at all, under my proposal.

> In additon, the user can easily compare the changes bewteen the package
> version shipped in the emacs tarball and the updated one fetched from
> ELPA, as the package layout is the same.

What for?  There should be NEWS in the new Org release.  Those more
details regarding the changes can always look in the Org Git repo.

> There are many more users of emacs than developers, so the design
> should be aimed at utility and convenience for users.

That's the main motivation for my proposal, indeed.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Andy Moreton]

On Fri 14 Oct 2016, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 14 Oct 2016 14:51:15 +0100
>> 
>> >> ~/.emacs.d/elpa/org/org
>> >> ~/.emacs.d/elpa/org/etc/ORG-NEWS
>> >> ~/.emacs.d/elpa/org/org.el
>> >> 
>> >> So, org-mode has to support two independent directory layouts.
>> >
>> > But Org already supports those two formats.  We don't require Org to
>> > do anything that it doesn't already do.  So staying with the current
>> > structure of lisp/ in the Emacs tree doesn't add any new requirements.
>> 
>> But that would do nothing to reduce the unnecessary and duplicated
>> packaging work.
>
> I'm not sure I understand what duplicated work you have in mind.  Org
> developers will still put Org on ELPA in the same directory structure
> they do now.  When Org is imported into Emacs, as part of building a
> release tarball or as part of building from the repository, each file
> will be put in its natural place in the Emacs source tree, either by
> package.el or by some other means, and will replace any previous Org
> files, if there were such.

It natural place in the emacs source tree should be the exact layout used
for ELPA, without any rearrangement. You proposal requires rearranging
the sources for the convenience of the few emacs develoeprs rather than
the manu emacs users. I do not think that is the right approach.

>> Keeping each package in ELPA format ensures that replacing the package
>> can be done easily, as everything is isolated in a single directory.
>
> I see no particular difficulties in putting several files into several
> directories, software can and does do that all the time.  It's not
> like we will be asking users to do this manually.

It is a pointless rearrangement from the package format, and makes emacs
core needlessly different from the package distribution. Bundling with
emacs core or distributing separately via ELPA should only concern how a
package is delivered, but should not change its content or layout.

>> There are many more users of emacs than developers, so the design
>> should be aimed at utility and convenience for users.
>
> That's the main motivation for my proposal, indeed.

Then you have lost me, as your proposal in this thread seems to be to
keeo the existing emacs directory structure, and make the layout of
packages bundled with emacs be different from the layout of the same
packages in ELPA.

The existing emacs source tree directory structure is sensible for a
monolithic project. However the whole idea here is to move away from a
monolithic structure, so the source tree contains only the emacs
core. Unmodified ELPA packages are imported into a packages directory
to be bundled for distribution.

    AndyM

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 14 Oct 2016 16:12:14 +0100
> 
> >> There are many more users of emacs than developers, so the design
> >> should be aimed at utility and convenience for users.
> >
> > That's the main motivation for my proposal, indeed.
> 
> Then you have lost me, as your proposal in this thread seems to be to
> keeo the existing emacs directory structure, and make the layout of
> packages bundled with emacs be different from the layout of the same
> packages in ELPA.
> 
> The existing emacs source tree directory structure is sensible for a
> monolithic project. However the whole idea here is to move away from a
> monolithic structure, so the source tree contains only the emacs
> core. Unmodified ELPA packages are imported into a packages directory
> to be bundled for distribution.

I guess we have different ideas of what will be left in the core,
then.  I think most of the stuff will be left in the core.

But in any case, having a separate sub-directory for every package,
like what we have on ELPA, makes very little sense for a structure
distributed in a release tarball.  You'd have many dozens of
subdirectories, each one with one or a handful of files.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> But in any case, having a separate sub-directory for every package, like
EZ> what we have on ELPA, makes very little sense for a structure distributed
EZ> in a release tarball. You'd have many dozens of subdirectories, each one
EZ> with one or a handful of files.

+1

It also occurred to me that we don't need a "mapping" file: We can impose the
constraint that any ELPA package to be included in the distribution use,
within its package, the same directory layout it would like overlaid into the
distribution.

I don't see why this issue is generating so much discussion. We've decided
we're not changing the directory structure for now. Supporting a single layout
in the final tarball is not hard. Why the push to cater to package.el?

If a user installs Emacs from the tarball, and then wishes to use Org-mode
From ELPA rather than the distribution, they'll do what they'd do today: Use
M-x package-install to install a newer version of Org-mode in their package
directory, shadowing the Org-mode we included in the distribution.

As far as I can tell, the only thing we need to support tarball ELPA is file
containing a list of packages, and an addition to "make dist" that copies
these packages into the distribution directory when building the tarball. Or
am I missing something?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Andy Moreton <andrewjmoreton@gmail.com> writes:
>> But Org already supports those two formats.  We don't require Org to
>> do anything that it doesn't already do.  So staying with the current
>> structure of lisp/ in the Emacs tree doesn't add any new requirements.
>
> But that would do nothing to reduce the unnecessary and duplicated
> packaging work.
>
> Keeping each package in ELPA format ensures that replacing the package
> can be done easily, as everything is isolated in a single directory. If
> the package is shipped in the emacs tarball and the user then upgrades
> to a newer version from ELPA, only the load path needs to change.


Yes, this is precisely my argument. Each elisp package should be
packaged in one and only one way and used in this way. Anything else
adds complexity for the developers of that package that is unfortunate.

Given that package.el provides a packaging solution that we already use,
adding it to the core build enables this simply and straightforwardly.


> In additon, the user can easily compare the changes bewteen the package
> version shipped in the emacs tarball and the updated one fetched from
> ELPA, as the package layout is the same.
>
> There are many more users of emacs than developers, so the design
> should be aimed at utility and convenience for users.

In this case, I am not sure this is so relevant; or, rather, it is
highly dependent on what you mean by "users". My belief is that the real
end users should see no difference at all. So the cost-benefit is
between the developers of individual packages, the developers of core
Emacs, and the developer of the build system.

My belief is that the first group win without cost from my proposal.
The second group, I think also win, especially if we can use some git
cleverness to make the core and tarball ELPA packages easy to update via
vc (i.e. some git submodule type of thing). And for the latter, it's not
clear; we have to offset getting package.el into the build system (which
I have proof of principle), against that of moving files around the
source tree, then building as usual.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> The existing emacs source tree directory structure is sensible for a
>> monolithic project. However the whole idea here is to move away from a
>> monolithic structure, so the source tree contains only the emacs
>> core. Unmodified ELPA packages are imported into a packages directory
>> to be bundled for distribution.
>
> I guess we have different ideas of what will be left in the core,
> then.  I think most of the stuff will be left in the core.
>
> But in any case, having a separate sub-directory for every package,
> like what we have on ELPA, makes very little sense for a structure
> distributed in a release tarball.  You'd have many dozens of
> subdirectories, each one with one or a handful of files.

My build system supports immediate sub-directories -- we can
subcategorise package.el format directories if we choose. But, yes, my
proposal would mean more subdirectories with a few files all aimed at
the same package, rather than a few directories with lots of files all
doing different things.

Why is having directories a bad thing? Actually, I think it's a good
thing.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> But in any case, having a separate sub-directory for every package, like
> EZ> what we have on ELPA, makes very little sense for a structure distributed
> EZ> in a release tarball. You'd have many dozens of subdirectories, each one
> EZ> with one or a handful of files.
>
> +1
>
> It also occurred to me that we don't need a "mapping" file: We can impose the
> constraint that any ELPA package to be included in the distribution use,
> within its package, the same directory layout it would like overlaid into the
> distribution.
>
> I don't see why this issue is generating so much discussion.

It's generating less discussion than the one about curly quotes, so I
think we are doing okay.

> We've decided we're not changing the directory structure for now.
> Supporting a single layout in the final tarball is not hard. Why the
> push to cater to package.el?

Because it's neat, simple and makes sense.


> If a user installs Emacs from the tarball, and then wishes to use Org-mode
> From ELPA rather than the distribution, they'll do what they'd do today: Use
> M-x package-install to install a newer version of Org-mode in their package
> directory, shadowing the Org-mode we included in the distribution.

Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
org. Now do, M-x load-library org-html. As you might expect org-html is
duly loaded.

This is unfortunate indeed because org-html is NOT in org any more. It
is still in the load-path though, and so still gets loaded. So, now we
have two versions of org loaded at the same time.

Of course, we could have package.el do cleverer things. It could realise
that org mode is also in core. It could therefore remove the core
installed org-mode directory from load-path. This would work for org, of
course, but only because org has its own subdirectory.

package.el could also check for existing autoloads. org-export-as-html,
for example, defined in org-html doesn't exist any more either, but, for
Emacs 24.3, its in the autoload file. Does package.el remove old
autoloads that are not shadowed? I don't know.

Alternatively, if org in core were in package.el format, package.el
could just not initialize the package in core. Never gets added to
load-path, autoloads never get loaded. package.el can use *exactly* the
same technique it uses when multiple version of org get installed into
~/.emacs.d/elpa.

> As far as I can tell, the only thing we need to support tarball ELPA is file
> containing a list of packages, and an addition to "make dist" that copies
> these packages into the distribution directory when building the tarball. Or
> am I missing something?

Each one in their own directory?

I think it's decision time. I am happy to carry on a little further with
the package.el based approach that I have outlined, fixing the one
significant issue with it and then I will stop. If you don't want to go
this way, that's fine.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Andy Moreton <andrewjmoreton@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 15 Oct 2016 12:51:22 +0100
> 
> Why is having directories a bad thing?

Because you introduce extra levels for no good reason.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Andy Moreton <andrewjmoreton@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 15 Oct 2016 13:01:23 +0100
> 
> > If a user installs Emacs from the tarball, and then wishes to use Org-mode
> > From ELPA rather than the distribution, they'll do what they'd do today: Use
> > M-x package-install to install a newer version of Org-mode in their package
> > directory, shadowing the Org-mode we included in the distribution.
> 
> Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
> org. Now do, M-x load-library org-html. As you might expect org-html is
> duly loaded.

How is this relevant to the issue at hand, though?  If you want to
solve this problem, all you need to is place all the ELPA directories
in load-path ahead of the standard ones, that's all.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Eli Zaretskii writes:
>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> 
>> > If a user installs Emacs from the tarball, and then wishes to use Org-mode
>> > From ELPA rather than the distribution, they'll do what they'd do today: Use
>> > M-x package-install to install a newer version of Org-mode in their package
>> > directory, shadowing the Org-mode we included in the distribution.
>> 
>> Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
>> org. Now do, M-x load-library org-html. As you might expect org-html is
>> duly loaded.
>
> How is this relevant to the issue at hand, though?  If you want to
> solve this problem, all you need to is place all the ELPA directories
> in load-path ahead of the standard ones, that's all.

How do you suppose the autoload for the no longer existing org-html will
be excised from the autoloads file in the old Emacs?  This _never_
worked with the "built-in packages" (which never were packages in the
first place) and package.el was apparently created with the assumption
that Emacs either started to use the packaging system internally as well
or extend it with additional functionality to support "built-in
packages" in Emacs' style.  The same mistkae happens if an autoloaded
function is moved to a different file in the ELPA package.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Sat, 15 Oct 2016 15:34:42 +0200
> 
> >> Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
> >> org. Now do, M-x load-library org-html. As you might expect org-html is
> >> duly loaded.
> >
> > How is this relevant to the issue at hand, though?  If you want to
> > solve this problem, all you need to is place all the ELPA directories
> > in load-path ahead of the standard ones, that's all.
> 
> How do you suppose the autoload for the no longer existing org-html will
> be excised from the autoloads file in the old Emacs?

Once again, how is this relevant to the directory structure of Lisp
files?  Can a particular structure solve this problem, where other
structures don't?

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Achim Gratz <Stromeko@nexgo.de>
>> Date: Sat, 15 Oct 2016 15:34:42 +0200
>> 
>> >> Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
>> >> org. Now do, M-x load-library org-html. As you might expect org-html is
>> >> duly loaded.
>> >
>> > How is this relevant to the issue at hand, though?  If you want to
>> > solve this problem, all you need to is place all the ELPA directories
>> > in load-path ahead of the standard ones, that's all.
>> 
>> How do you suppose the autoload for the no longer existing org-html will
>> be excised from the autoloads file in the old Emacs?
>
> Once again, how is this relevant to the directory structure of Lisp
> files?  Can a particular structure solve this problem, where other
> structures don't?

Yes. Package.el solves this problem quite nicely. We have:

~/.emacs.d/elpa/org-mode-1.6/org-html.el
~/.emacs.d/elpa/org-mode-1.8/ox-html.el

package.el solves the problem that org-html.el no longer exists simply
by not adding org-mode-1.6 to the load path. Nor does it load
"org-autoloads.el". So, even though org-html.el is still around on the
hard drive, it is ignored by emacs.

Now, as it happens, with the current directory organisation of Emacs, we
can solve the problem anyway, by doing the same trick in core. But,
currently, we don't.

And this could only work for org.el *because* org is in it's own
directory with its own autoloads file. For seq.el, for instance, it will
not work.

IIUC, in fact it's the org-mode folks actually view this as a
significant *disadvantage* to having org-mode in core. It's only because
this is the only way to get it into the tarball that they do it. Achim,
do I have this right?

There are a few other things that package.el does, which it would be
shame to loose. It checks dependencies for instance. So, say, I
installed assess.el into Emacs core using package.el as I suggest, and
it requires a newer version of seq.el than is available, package.el will
bitch about this very early.

We could achieve all of this without package.el, with copying and
checking. But why? package.el already does all of this.

Bottom line: package.el is a better way of handling packages than core
which is a legacy. Moving wholesale to it is probably not worth the
effort (at least not in the short term). But supporting it in core
definately is.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Achim Gratz <Stromeko@nexgo.de>,  emacs-devel@gnu.org
> Date: Sat, 15 Oct 2016 15:53:33 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Achim Gratz <Stromeko@nexgo.de>
> >> Date: Sat, 15 Oct 2016 15:34:42 +0200
> >> 
> >> >> Except that it doesn't. Try this. Take Emacs 24.3, M-x package-install
> >> >> org. Now do, M-x load-library org-html. As you might expect org-html is
> >> >> duly loaded.
> >> >
> >> > How is this relevant to the issue at hand, though?  If you want to
> >> > solve this problem, all you need to is place all the ELPA directories
> >> > in load-path ahead of the standard ones, that's all.
> >> 
> >> How do you suppose the autoload for the no longer existing org-html will
> >> be excised from the autoloads file in the old Emacs?
> >
> > Once again, how is this relevant to the directory structure of Lisp
> > files?  Can a particular structure solve this problem, where other
> > structures don't?
> 
> Yes. Package.el solves this problem quite nicely. We have:
> 
> ~/.emacs.d/elpa/org-mode-1.6/org-html.el
> ~/.emacs.d/elpa/org-mode-1.8/ox-html.el
> 
> package.el solves the problem that org-html.el no longer exists simply
> by not adding org-mode-1.6 to the load path. Nor does it load
> "org-autoloads.el". So, even though org-html.el is still around on the
> hard drive, it is ignored by emacs.

The same can be done with any other directory structure, in two steps:
(1) put the new directory first in load-path, and (2) don't load
org-autoloads.el.  Right?

> Now, as it happens, with the current directory organisation of Emacs, we
> can solve the problem anyway, by doing the same trick in core. But,
> currently, we don't.

That's exactly what I meant: we can do this with any directory
structure.  IOW the directory structure is not relevant to this issue.

> And this could only work for org.el *because* org is in it's own
> directory with its own autoloads file. For seq.el, for instance, it will
> not work.

So one solution would be to provide a separate autoloads file for each
package that lives on ELPA.  That again has nothing to do with the
directory structure.  Right?  (There could also be other solutions.)

> IIUC, in fact it's the org-mode folks actually view this as a
> significant *disadvantage* to having org-mode in core.

What is "this" that the Org folks see as a disadvantage?

> There are a few other things that package.el does, which it would be
> shame to loose. It checks dependencies for instance. So, say, I
> installed assess.el into Emacs core using package.el as I suggest, and
> it requires a newer version of seq.el than is available, package.el will
> bitch about this very early.

I don't see why we'd have to lose this.  No one said we are throwing
away package.el.  We are talking about _extending_ it.

> Bottom line: package.el is a better way of handling packages than core
> which is a legacy. Moving wholesale to it is probably not worth the
> effort (at least not in the short term). But supporting it in core
> definately is.

We need to adapt package.el to the new needs.  It was not written with
these goals in mind, so it needs to learn new tricks.  Throwing it
away is not acceptable, but neither is blindly accepting its current
assumptions, which were not designed for the use case we are
discussing.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Eli Zaretskii writes:
>> How do you suppose the autoload for the no longer existing org-html will
>> be excised from the autoloads file in the old Emacs?
>
> Once again, how is this relevant to the directory structure of Lisp
> files?  Can a particular structure solve this problem, where other
> structures don't?

No, it is not directly related to the directory structure.  But Emacs
does collect all first-level autoloads for all elisp ("package" moniker
or not) into a single file, with no provisions to deactivate them when a
newer version of the same in-core package gets installed again.  There
are probably many different ways to tease this apart, but treating an
in-core package no different than an out-of-core package, apart from the
fact that it comes with the tarball and gets installed into a different
tree has the advantage that it doesn't need completely new mechanisms to
support it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Sat, 15 Oct 2016 19:08:35 +0200
> 
> No, it is not directly related to the directory structure.  But Emacs
> does collect all first-level autoloads for all elisp ("package" moniker
> or not) into a single file, with no provisions to deactivate them when a
> newer version of the same in-core package gets installed again.

This will have to be solved, of course.  We already have some
foo-autoloads.el files for some packages; perhaps that would be the
solution here as well.

> There are probably many different ways to tease this apart, but
> treating an in-core package no different than an out-of-core
> package, apart from the fact that it comes with the tarball and gets
> installed into a different tree has the advantage that it doesn't
> need completely new mechanisms to support it.

I don't think we will be able to treat these two classes of packages
the same.  I'm quite sure more and more reasons will pop up as we go.
So I believe we will have special treatment for ELPA packages anyway.
I just think that the directory in which they live as part of the
Emacs tree doesn't have to be part of that special treatment.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Phillip Lord writes:
> And this could only work for org.el *because* org is in it's own
> directory with its own autoloads file. For seq.el, for instance, it will
> not work.

No, some autoloads still get collected into the top-level autoloads
file.  The same two-layer mechanism is used by calc.el, IIRC.

> IIUC, in fact it's the org-mode folks actually view this as a
> significant *disadvantage* to having org-mode in core. It's only because
> this is the only way to get it into the tarball that they do it. Achim,
> do I have this right?

Please talk to the current Org maintainers about this, I was unable to
spend much time on Org during the last year or so.  In any case, I don't
think this should be viewed as an issue with Org, that's certain to rub
someone the wrong way.

My personal biew is that anything that shows up as a "built-in package"
should behave just like it had been installed by package.el from the POV
of a user.  So, a user should be able to cleanly deactivate or upgrade
each such package via package.el.

> There are a few other things that package.el does, which it would be
> shame to loose. It checks dependencies for instance. So, say, I
> installed assess.el into Emacs core using package.el as I suggest, and
> it requires a newer version of seq.el than is available, package.el will
> bitch about this very early.

Paackage.el has it's own limitations which unfortunately are also
highlighted by Org.  Some of the files that go into ELPA Org are
actually generated by an upstream makefile so that the tarball that ELPA
distributes can be successfully installed within those limitations.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: John Wiegley <jwiegley@gmail.com>,  a.s@realize.ch,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Fri, 14 Oct 2016 09:25:13 +0100
>> 
>> >> I think we should *not* give up the standard file structure. At least not now.
>> >> Using ELPA should feel "first class" for those authors contributing packages
>> >> that are to become part of the standard distribution.
>> >
>> > 100% agreement.
>> 
>> Can you tell me why, though? Drew complained about grepability. What are
>> the other reasons?
>
> The structure of the Emacs lisp/ directory is well-thought and exists
> for many years with only minor changes.  It has some underlying logic,
> which allows one in most cases to know where a certain package lives.
> This is important not just for grepping, but also for visiting the
> files and any operation that requires its full file name.  Having some
> files outside of this structure will make working with those files
> more annoying.

The lisp directory structure is, to my mind, pretty confused. We have
"emacs-lisp", but one of emacs-lisp's more distinctive features, custom,
is not in it. Likewise, tree-widget. We have "emacs-lisp", "progmodes"
and so on which are defined after function, and "obsolete" which is
defined after status. We have "text-modes" and "org". We have "mail" and
"gnus" (although some of gnus is in net). Likewise mh. Likewise
international, language and leim.

Still, I agree that changing things is always a pain.

Phil

Re: [Emacs-diffs] feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: jwiegley@gmail.com,  a.s@realize.ch,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 15 Oct 2016 18:41:15 +0100
> 
> > The structure of the Emacs lisp/ directory is well-thought and exists
> > for many years with only minor changes.  It has some underlying logic,
> > which allows one in most cases to know where a certain package lives.
> > This is important not just for grepping, but also for visiting the
> > files and any operation that requires its full file name.  Having some
> > files outside of this structure will make working with those files
> > more annoying.
> 
> The lisp directory structure is, to my mind, pretty confused. We have
> "emacs-lisp", but one of emacs-lisp's more distinctive features, custom,
> is not in it. Likewise, tree-widget. We have "emacs-lisp", "progmodes"
> and so on which are defined after function, and "obsolete" which is
> defined after status. We have "text-modes" and "org". We have "mail" and
> "gnus" (although some of gnus is in net). Likewise mh. Likewise
> international, language and leim.

I could give reasons for most each one of those.  But the structure is
not sacred; we can change it if we decide so.  The important thing,
though, is that there _are_ reasons for the structure, and they aren't
just that each package has its own directory.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> I think it's decision time. I am happy to carry on a little further with
PL> the package.el based approach that I have outlined, fixing the one
PL> significant issue with it and then I will stop. If you don't want to go
PL> this way, that's fine.

At this time, I don't want to go "full package.el".  However, I'd like to
include it, since it's key to how users interact with ELPA-based packages.  I
think Eli said it best:

EZ> We need to adapt package.el to the new needs. It was not written with
EZ> these goals in mind, so it needs to learn new tricks. Throwing it away is
EZ> not acceptable, but neither is blindly accepting its current assumptions,
EZ> which were not designed for the use case we are discussing.

So let's not move to "directory per package", but let's do support "properly
upgrading a package that came with the distribution".

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Andy Moreton <andrewjmoreton@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 17 Oct 2016 16:09:15 -0700
> 
> PL> I think it's decision time. I am happy to carry on a little further with
> PL> the package.el based approach that I have outlined, fixing the one
> PL> significant issue with it and then I will stop. If you don't want to go
> PL> this way, that's fine.

Sorry, I don't understand the kind of decision that is being
requested.  I understand that one alternative is that you "carry on a
little further" with your approach, although I'm vague about the
details of that, or what is your goal.  But the other alternative is
entirely unclear to me.  What is it?

> At this time, I don't want to go "full package.el".  However, I'd like to
> include it, since it's key to how users interact with ELPA-based packages.

I don't think I understand what does "full package.el" mean.  However,
package.el is already included, it's in lisp/emacs-lisp/.

> I think Eli said it best:
> 
> EZ> We need to adapt package.el to the new needs. It was not written with
> EZ> these goals in mind, so it needs to learn new tricks. Throwing it away is
> EZ> not acceptable, but neither is blindly accepting its current assumptions,
> EZ> which were not designed for the use case we are discussing.
> 
> So let's not move to "directory per package", but let's do support "properly
> upgrading a package that came with the distribution".

That's obviously fine with me ;-)  I just am not sure this is one of
the alternatives that Phillip considers.

Thanks.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> package.el solves the problem that org-html.el no longer exists simply
>> by not adding org-mode-1.6 to the load path. Nor does it load
>> "org-autoloads.el". So, even though org-html.el is still around on the
>> hard drive, it is ignored by emacs.
>
> The same can be done with any other directory structure, in two steps:
> (1) put the new directory first in load-path, and (2) don't load
> org-autoloads.el.  Right?

Apologies if I have replied to this already -- getting confused.

Yes, it could be.


>> Now, as it happens, with the current directory organisation of Emacs, we
>> can solve the problem anyway, by doing the same trick in core. But,
>> currently, we don't.
>
> That's exactly what I meant: we can do this with any directory
> structure.  IOW the directory structure is not relevant to this issue.

I think that this is incorrect. In this case, we can do this with the
directory structure that we have at the moment, and for org-mode,
because org-mode is a) in a directory of it's own and b) has it's own
loaddefs. This is not true for many of the files in ./lisp.

The point here is that load-path is *directory* based. As it stands, we
cannot exclude some files in one directory.


>> And this could only work for org.el *because* org is in it's own
>> directory with its own autoloads file. For seq.el, for instance, it will
>> not work.
>
> So one solution would be to provide a separate autoloads file for each
> package that lives on ELPA.  That again has nothing to do with the
> directory structure.  Right?  (There could also be other solutions.)

I think that it is evident that there could be other solutions. Emacs is
Turing complete so any computable solution that you can image is
possible. The question is which is easiest.

package.el already does all of this.


>> There are a few other things that package.el does, which it would be
>> shame to loose. It checks dependencies for instance. So, say, I
>> installed assess.el into Emacs core using package.el as I suggest, and
>> it requires a newer version of seq.el than is available, package.el will
>> bitch about this very early.
>
> I don't see why we'd have to lose this.  No one said we are throwing
> away package.el.  We are talking about _extending_ it.

Just so that we can keep a particular directory structure?


>> Bottom line: package.el is a better way of handling packages than core
>> which is a legacy. Moving wholesale to it is probably not worth the
>> effort (at least not in the short term). But supporting it in core
>> definately is.
>
> We need to adapt package.el to the new needs.  It was not written with
> these goals in mind, so it needs to learn new tricks.  Throwing it
> away is not acceptable, but neither is blindly accepting its current
> assumptions, which were not designed for the use case we are
> discussing.

No, but it's current behaviour makes a lot of sense to me and is, I
think, better than the monolithic structure we have in ./lisp. Yes, it
needs to be taught new tricks and it needs to be -- it can't cope with
texinfo files, for instance.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Achim Gratz <Stromeko@nexgo.de> writes:

> Phillip Lord writes:
>> And this could only work for org.el *because* org is in it's own
>> directory with its own autoloads file. For seq.el, for instance, it will
>> not work.
>
> No, some autoloads still get collected into the top-level autoloads
> file.  The same two-layer mechanism is used by calc.el, IIRC.

Most of them are in loaddefs. Quite a few "packages" have their own.


>> IIUC, in fact it's the org-mode folks actually view this as a
>> significant *disadvantage* to having org-mode in core. It's only because
>> this is the only way to get it into the tarball that they do it. Achim,
>> do I have this right?
>
> Please talk to the current Org maintainers about this, I was unable to
> spend much time on Org during the last year or so.  In any case, I don't
> think this should be viewed as an issue with Org, that's certain to rub
> someone the wrong way.

Apologies if it read like this; I don't think this is a problem with
org, I think it's a problem with Emacs.


> My personal biew is that anything that shows up as a "built-in package"
> should behave just like it had been installed by package.el from the POV
> of a user.  So, a user should be able to cleanly deactivate or upgrade
> each such package via package.el.
>
>> There are a few other things that package.el does, which it would be
>> shame to loose. It checks dependencies for instance. So, say, I
>> installed assess.el into Emacs core using package.el as I suggest, and
>> it requires a newer version of seq.el than is available, package.el will
>> bitch about this very early.
>
> Paackage.el has it's own limitations which unfortunately are also
> highlighted by Org.  Some of the files that go into ELPA Org are
> actually generated by an upstream makefile so that the tarball that ELPA
> distributes can be successfully installed within those limitations.

Do tell. Which files are generated?

Using package.el in core would force us to fix these things, I think,
which is a good thing!

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> No, it is not directly related to the directory structure.  But Emacs
>> does collect all first-level autoloads for all elisp ("package" moniker
>> or not) into a single file, with no provisions to deactivate them when a
>> newer version of the same in-core package gets installed again.
>
> This will have to be solved, of course.  We already have some
> foo-autoloads.el files for some packages; perhaps that would be the
> solution here as well.


I agree with this, although I'd like to keep generation of the loaddefs
as simple as possible for a variety of reasons; each requires a separate
invocation during build, and IIRC, this is part of the build which is
not parallelizable.


>> There are probably many different ways to tease this apart, but
>> treating an in-core package no different than an out-of-core
>> package, apart from the fact that it comes with the tarball and gets
>> installed into a different tree has the advantage that it doesn't
>> need completely new mechanisms to support it.
>
> I don't think we will be able to treat these two classes of packages
> the same.  I'm quite sure more and more reasons will pop up as we go.


I am trying and we will see.

> So I believe we will have special treatment for ELPA packages anyway.
> I just think that the directory in which they live as part of the
> Emacs tree doesn't have to be part of that special treatment.

We have to something special ELPA packages and their directory
structure, whether that it is working out how to move them into the
existing structure, or having some packages build and installed as part
of the normal build process.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 11:52:17 +0100
> 
> > That's exactly what I meant: we can do this with any directory
> > structure.  IOW the directory structure is not relevant to this issue.
> 
> I think that this is incorrect. In this case, we can do this with the
> directory structure that we have at the moment, and for org-mode,
> because org-mode is a) in a directory of it's own and b) has it's own
> loaddefs. This is not true for many of the files in ./lisp.

When I said "we can do this" I didn't mean "without any changes" or
"for free".  We will have to make changes, both in how loaddefs are
generated, and in the build.  My point is that these changes are not
difficult, since we already do that for several distinct packages in
the core.  We just need to do that for more packages, that's all.

> The point here is that load-path is *directory* based. As it stands, we
> cannot exclude some files in one directory.

I don't see any need to exclude files.  If a newer version of a
package is installed outside of the Emacs lisp/ tree, the directory of
that package needs to be earlier in load-path than the main lisp/
directory and its subdirectories.  Again, not hard to arrange.

> > So one solution would be to provide a separate autoloads file for each
> > package that lives on ELPA.  That again has nothing to do with the
> > directory structure.  Right?  (There could also be other solutions.)
> 
> I think that it is evident that there could be other solutions. Emacs is
> Turing complete so any computable solution that you can image is
> possible. The question is which is easiest.

They are all easy.

> package.el already does all of this.

Once again, package.el in its current form was never meant to be used
in the capacity we are talking about.  So using it without changes for
the job we are discussing is wrong, because it would mean to subdue
our directory organization to decisions made for a completely
different use case.

In any case, such a solution was already rejected, both by John and
myself.  So let's not argue about this anymore; let's instead see
which changes are needed in package.el to support the preferred
directory structure in core, even when some of the bundled packages
live on ELPA.

> >> There are a few other things that package.el does, which it would be
> >> shame to loose. It checks dependencies for instance. So, say, I
> >> installed assess.el into Emacs core using package.el as I suggest, and
> >> it requires a newer version of seq.el than is available, package.el will
> >> bitch about this very early.
> >
> > I don't see why we'd have to lose this.  No one said we are throwing
> > away package.el.  We are talking about _extending_ it.
> 
> Just so that we can keep a particular directory structure?

Yes!

> > We need to adapt package.el to the new needs.  It was not written with
> > these goals in mind, so it needs to learn new tricks.  Throwing it
> > away is not acceptable, but neither is blindly accepting its current
> > assumptions, which were not designed for the use case we are
> > discussing.
> 
> No, but it's current behaviour makes a lot of sense to me and is, I
> think, better than the monolithic structure we have in ./lisp.

Others, including myself, disagree.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Achim Gratz <Stromeko@nexgo.de>,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 11:59:20 +0100
> 
> > This will have to be solved, of course.  We already have some
> > foo-autoloads.el files for some packages; perhaps that would be the
> > solution here as well.
> 
> I agree with this, although I'd like to keep generation of the loaddefs
> as simple as possible for a variety of reasons; each requires a separate
> invocation during build, and IIRC, this is part of the build which is
> not parallelizable.

Why do you think it isn't parallelizable?  I think it is, and we
already do run these targets in parallel.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Andy Moreton <andrewjmoreton@gmail.com>,  emacs-devel@gnu.org
>> Date: Mon, 17 Oct 2016 16:09:15 -0700
>> 
>> PL> I think it's decision time. I am happy to carry on a little further with
>> PL> the package.el based approach that I have outlined, fixing the one
>> PL> significant issue with it and then I will stop. If you don't want to go
>> PL> this way, that's fine.
>
> Sorry, I don't understand the kind of decision that is being
> requested.  I understand that one alternative is that you "carry on a
> little further" with your approach, although I'm vague about the
> details of that, or what is your goal.

Complete the code that I have started. Three goals:

 - Use package.el to build (compile, generate autoloads) packages in
   during the build
 - Use package.el to initialize and load these during startup
 - Support testing of these packages during build.

With a secondary aim of:

 - Enable a built emacs to build and test all packages in ELPA whether
   they are core/tarball or not.

> But the other alternative is entirely unclear to me. What is it?


Erm, don't do this.

Or write something such as you have been talking about, by copying bits
of ELPA into core, otherwise (presumably) leaving the build alone.


>> At this time, I don't want to go "full package.el".  However, I'd like to
>> include it, since it's key to how users interact with ELPA-based packages.
>
> I don't think I understand what does "full package.el" mean.  However,
> package.el is already included, it's in lisp/emacs-lisp/.

I presume John means what I have discussed above.

>
>> I think Eli said it best:
>> 
>> EZ> We need to adapt package.el to the new needs. It was not written with
>> EZ> these goals in mind, so it needs to learn new tricks. Throwing it away is
>> EZ> not acceptable, but neither is blindly accepting its current assumptions,
>> EZ> which were not designed for the use case we are discussing.
>> 
>> So let's not move to "directory per package", but let's do support "properly
>> upgrading a package that came with the distribution".
>
> That's obviously fine with me ;-)  I just am not sure this is one of
> the alternatives that Phillip considers.

It's obviously difficult for me to think of better ways of achieving
this than my current proposal. There are probably worse ways of
achieving it, I am sure.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> I think that this is incorrect. In this case, we can do this with the
>> directory structure that we have at the moment, and for org-mode,
>> because org-mode is a) in a directory of it's own and b) has it's own
>> loaddefs. This is not true for many of the files in ./lisp.
>
> When I said "we can do this" I didn't mean "without any changes" or
> "for free".  We will have to make changes, both in how loaddefs are
> generated, and in the build.  My point is that these changes are not
> difficult, since we already do that for several distinct packages in
> the core.  We just need to do that for more packages, that's all.

Yes.

>
>> The point here is that load-path is *directory* based. As it stands, we
>> cannot exclude some files in one directory.
>
> I don't see any need to exclude files.  If a newer version of a
> package is installed outside of the Emacs lisp/ tree, the directory of
> that package needs to be earlier in load-path than the main lisp/
> directory and its subdirectories.  Again, not hard to arrange.

No, this doesn't work. The point is with the new exporter, org
introduced new files. The new ox-html.el would not shadow the old
org-html.el, however you organised the path. In otherwords, the org
package changed the features that it provided over time.

More generally, you might also concieve of a situation where a new
package replaces another. Same 


>> package.el already does all of this.
>
> Once again, package.el in its current form was never meant to be used
> in the capacity we are talking about.  So using it without changes for
> the job we are discussing is wrong, because it would mean to subdue
> our directory organization to decisions made for a completely
> different use case.
> 
> In any case, such a solution was already rejected, both by John and
> myself.  So let's not argue about this anymore; let's instead see
> which changes are needed in package.el to support the preferred
> directory structure in core, even when some of the bundled packages
> live on ELPA.


Okay. I may complete my branch anyway, just for completeness sake.



>> > We need to adapt package.el to the new needs.  It was not written with
>> > these goals in mind, so it needs to learn new tricks.  Throwing it
>> > away is not acceptable, but neither is blindly accepting its current
>> > assumptions, which were not designed for the use case we are
>> > discussing.
>> 
>> No, but it's current behaviour makes a lot of sense to me and is, I
>> think, better than the monolithic structure we have in ./lisp.
>
> Others, including myself, disagree.

Yes, that is apparent. No worries.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> I agree with this, although I'd like to keep generation of the loaddefs
>> as simple as possible for a variety of reasons; each requires a separate
>> invocation during build, and IIRC, this is part of the build which is
>> not parallelizable.
>
> Why do you think it isn't parallelizable?  I think it is, and we
> already do run these targets in parallel.

Just from memory of watching the build scroll pass on screen.

I may be wrong.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Andy Moreton]

On Mon 17 Oct 2016, John Wiegley wrote:

>>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> PL> I think it's decision time. I am happy to carry on a little further with
> PL> the package.el based approach that I have outlined, fixing the one
> PL> significant issue with it and then I will stop. If you don't want to go
> PL> this way, that's fine.
>
> At this time, I don't want to go "full package.el".  However, I'd like to
> include it, since it's key to how users interact with ELPA-based packages.  I
> think Eli said it best:
>
> EZ> We need to adapt package.el to the new needs. It was not written with
> EZ> these goals in mind, so it needs to learn new tricks. Throwing it away is
> EZ> not acceptable, but neither is blindly accepting its current assumptions,
> EZ> which were not designed for the use case we are discussing.
>
> So let's not move to "directory per package", but let's do support "properly
> upgrading a package that came with the distribution".

That goal seems much easier to achive if each ELPA-based package is
isolated in its own directory, and all dependencies from core-emacs to
such packages are removed (including autoloads).

    AndyM

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 14:33:50 +0100
> 
> >> The point here is that load-path is *directory* based. As it stands, we
> >> cannot exclude some files in one directory.
> >
> > I don't see any need to exclude files.  If a newer version of a
> > package is installed outside of the Emacs lisp/ tree, the directory of
> > that package needs to be earlier in load-path than the main lisp/
> > directory and its subdirectories.  Again, not hard to arrange.
> 
> No, this doesn't work. The point is with the new exporter, org
> introduced new files. The new ox-html.el would not shadow the old
> org-html.el, however you organised the path. In otherwords, the org
> package changed the features that it provided over time.

Look, this discussion will go nowhere constructive if you change the
subject all the time.  You talked about load-path, and our inability
to exclude files in directories on load-path.  Clearly, this only
matters when Emacs wants to _load_ the file in question, and for that
the order of directories in load-path is all that counts.

Now you are evidently talking about a file that was already loaded,
and then the user upgraded the package from which that file comes, and
wants the new version to become effective without restarting Emacs,
which is where the old features already loaded get in the way because
the user wants their new versions.  Obviously, this has nothing to do
with load-path, right?

> More generally, you might also concieve of a situation where a new
> package replaces another.

If package.el already knows how to unload the old features and load
the new ones, it will continue doing this for any package, whether in
or out of core.  Right?

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 14:37:18 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> I agree with this, although I'd like to keep generation of the loaddefs
> >> as simple as possible for a variety of reasons; each requires a separate
> >> invocation during build, and IIRC, this is part of the build which is
> >> not parallelizable.
> >
> > Why do you think it isn't parallelizable?  I think it is, and we
> > already do run these targets in parallel.
> 
> Just from memory of watching the build scroll pass on screen.

Well, my memory is wrong.

> I may be wrong.

One of us is.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: John Wiegley <jwiegley@gmail.com>,  andrewjmoreton@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 14:01:52 +0100
> 
> >> PL> I think it's decision time. I am happy to carry on a little further with
> >> PL> the package.el based approach that I have outlined, fixing the one
> >> PL> significant issue with it and then I will stop. If you don't want to go
> >> PL> this way, that's fine.
> >
> > Sorry, I don't understand the kind of decision that is being
> > requested.  I understand that one alternative is that you "carry on a
> > little further" with your approach, although I'm vague about the
> > details of that, or what is your goal.
> 
> Complete the code that I have started. Three goals:
> 
>  - Use package.el to build (compile, generate autoloads) packages in
>    during the build
>  - Use package.el to initialize and load these during startup
>  - Support testing of these packages during build.
> 
> With a secondary aim of:
> 
>  - Enable a built emacs to build and test all packages in ELPA whether
>    they are core/tarball or not.

Can this be done while keeping the current structure under lisp/
intact and putting packages from ELPA under lisp/ ?  If yes, I think
this is what we all want.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> Date: Tue, 18 Oct 2016 17:48:27 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: Stromeko@nexgo.de, emacs-devel@gnu.org
> 
> Well, my memory is wrong.

I meant my memory is different, sorry.

Sometimes my fingers work ahead of my head.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
>> Date: Tue, 18 Oct 2016 14:33:50 +0100
>> 
>> >> The point here is that load-path is *directory* based. As it stands, we
>> >> cannot exclude some files in one directory.
>> >
>> > I don't see any need to exclude files.  If a newer version of a
>> > package is installed outside of the Emacs lisp/ tree, the directory of
>> > that package needs to be earlier in load-path than the main lisp/
>> > directory and its subdirectories.  Again, not hard to arrange.
>> 
>> No, this doesn't work. The point is with the new exporter, org
>> introduced new files. The new ox-html.el would not shadow the old
>> org-html.el, however you organised the path. In otherwords, the org
>> package changed the features that it provided over time.
>
> Look, this discussion will go nowhere constructive if you change the
> subject all the time.  You talked about load-path, and our inability
> to exclude files in directories on load-path.  Clearly, this only
> matters when Emacs wants to _load_ the file in question, and for that
> the order of directories in load-path is all that counts.


*shrugs*. I'm doing my best to be as clear as I can, I do not think I am
 changing the subject.


> Now you are evidently talking about a file that was already loaded,
> and then the user upgraded the package from which that file comes, and
> wants the new version to become effective without restarting Emacs,
> which is where the old features already loaded get in the way because
> the user wants their new versions.  Obviously, this has nothing to do
> with load-path, right?

No.

Consider this scenario.

Situation One:

Org-mode comes with a file called org-html. Org-mode v1 is distributed
with core emacs. Org-mode v2 is released, and user installs from ELPA.
Org-mode v2 is before Org-mode v1. The world is well.


Situation Two:

Org-mode v1 comes with a file called org-html. v1 is distributed with
core emacs. Org-mode v2, however, no longer has a file called org-html,
but does have a file called ox-html. Although org-mode v2 comes in the
path before org-mode v1, it is still possible to load org-html.


Situation Three:

Org-mode v1 comes with org-html. v1 is distributed through ELPA.
Org-mode v2 comes with ox-html. We upgrade v1 to v2, and v1 is removed
from the path. org-html is no longer loadable.


I mention this because it's happened to me. I was happily loaded
org-html even though I had updated org mode to latest in ELPA because I
didn't know.

We can solve Situation Two by simply excluding the org-mode v1 directory
for load path, but this option is only open to us because org-mode is in
its own directory.

My conclusion: having org-mode its own directory is a good thing. By
extrapolation, therefore, having most or all packages in their own
directory would be a good thing.


>> More generally, you might also concieve of a situation where a new
>> package replaces another.
>
> If package.el already knows how to unload the old features and load
> the new ones, it will continue doing this for any package, whether in
> or out of core.  Right?

With the example that I have given, this fails at the moment for
org-mode, specificially because org-mode is in core.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: John Wiegley <jwiegley@gmail.com>,  andrewjmoreton@gmail.com,  emacs-devel@gnu.org
>> Date: Tue, 18 Oct 2016 14:01:52 +0100
>> 
>> >> PL> I think it's decision time. I am happy to carry on a little further with
>> >> PL> the package.el based approach that I have outlined, fixing the one
>> >> PL> significant issue with it and then I will stop. If you don't want to go
>> >> PL> this way, that's fine.
>> >
>> > Sorry, I don't understand the kind of decision that is being
>> > requested.  I understand that one alternative is that you "carry on a
>> > little further" with your approach, although I'm vague about the
>> > details of that, or what is your goal.
>> 
>> Complete the code that I have started. Three goals:
>> 
>>  - Use package.el to build (compile, generate autoloads) packages in
>>    during the build
>>  - Use package.el to initialize and load these during startup
>>  - Support testing of these packages during build.
>> 
>> With a secondary aim of:
>> 
>>  - Enable a built emacs to build and test all packages in ELPA whether
>>    they are core/tarball or not.
>
> Can this be done while keeping the current structure under lisp/
> intact and putting packages from ELPA under lisp/ ?  If yes, I think
> this is what we all want.


I can easily do:


EMACS/lisp
EMACS/lisp/lots-of-files.el
EMACS/lisp/textmodes/
EMACS/lisp/progmodes
EMACS/lisp/packages
EMACS/lisp/packages/package-1
EMACS/lisp/packages/package-2
EMACS/lisp/packages/package-3


This adds a little complexity to the build part of the make file; in
return, it leaves the install part of the make file untouched. It also
requires a package.el to be extended slightly to extend the current
initialization process.

I could less easily do:

EMACS/lisp
EMACS/lisp/lots-of-files.el
EMACS/lisp/textmodes/package-1
EMACS/lisp/progmodes/package-2
EMACS/lisp/emacs-lisp/package-3


This requires updating the current build process to exclude multiple
directories, and the package.el section of my branch to include multiple
directories. I also dislike this, because I think it mixes up files of
different kinds in the tree (that is files which are in place in the
Emacs git repo, and those which are not).

Finally, it would no doubt be possible to do:

EMACS/lisp
EMACS/lisp/textmodes/package-1a.el
EMACS/lisp/textmodes/package-1b.el
EMACS/lisp/textmodes/package-1c.el
EMACS/lisp/progmodes/package-2.el
EMACS/lisp/emacs-lisp/package-3.el

This would need no use at all of package.el, but would require something
to get the files into the right place and should leave the build
otherwise untouched. Superficially, this is the simplest. However, in
practice, I think it's the most complex and least convienient, for
reasons I have outlined and documented.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 17:43:11 +0100
> 
> Org-mode v1 comes with a file called org-html. v1 is distributed with
> core emacs. Org-mode v2, however, no longer has a file called org-html,
> but does have a file called ox-html. Although org-mode v2 comes in the
> path before org-mode v1, it is still possible to load org-html.

That just means installing a new version needs to remove the previous
version's files first, that's all.  It has nothing to do with
load-path.

> Org-mode v1 comes with org-html. v1 is distributed through ELPA.
> Org-mode v2 comes with ox-html. We upgrade v1 to v2, and v1 is removed
> from the path. org-html is no longer loadable.

org-html doesn't need to be loadable if it doesn't exist in the new
version.  Again, nothing to do with load-path.

> My conclusion: having org-mode its own directory is a good thing. By
> extrapolation, therefore, having most or all packages in their own
> directory would be a good thing.

The extrapolation is invalid, because while Org is a very large
package, most other packages aren't.

> > If package.el already knows how to unload the old features and load
> > the new ones, it will continue doing this for any package, whether in
> > or out of core.  Right?
> 
> With the example that I have given, this fails at the moment for
> org-mode, specificially because org-mode is in core.

I don't understand why it fails, but if it's a missing feature in
package.el or in the infrastructure, we need to add that, in order to
move to having bundled packages on ELPA.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: jwiegley@gmail.com,  andrewjmoreton@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 17:59:28 +0100
> 
> >>  - Enable a built emacs to build and test all packages in ELPA whether
> >>    they are core/tarball or not.
> >
> > Can this be done while keeping the current structure under lisp/
> > intact and putting packages from ELPA under lisp/ ?  If yes, I think
> > this is what we all want.
> 
> 
> I can easily do:
> 
> 
> EMACS/lisp
> EMACS/lisp/lots-of-files.el
> EMACS/lisp/textmodes/
> EMACS/lisp/progmodes
> EMACS/lisp/packages
> EMACS/lisp/packages/package-1
> EMACS/lisp/packages/package-2
> EMACS/lisp/packages/package-3

As already discussed, this is not what we want.

> I could less easily do:
> 
> EMACS/lisp
> EMACS/lisp/lots-of-files.el
> EMACS/lisp/textmodes/package-1
> EMACS/lisp/progmodes/package-2
> EMACS/lisp/emacs-lisp/package-3

This is what we want.  IOW, we want foo.el to live in the same
directory it is now, even if its Git repository will be on ELPA.

> This requires updating the current build process to exclude multiple
> directories

Why do we need to exclude directories during the build?  And what does
"exclude" mean in this context?

> I also dislike this, because I think it mixes up files of different
> kinds in the tree (that is files which are in place in the Emacs git
> repo, and those which are not).

There are no different kinds.  They are all of the same kind: Lisp
packages that are bundled with a release tarball.  The fact that some
of them are in a separate Git repo  doesn't make them a different
kind.  We want the move to ELPA to be transparent as far as the Emacs
lisp/ directory structure is concerned.

> Finally, it would no doubt be possible to do:
> 
> EMACS/lisp
> EMACS/lisp/textmodes/package-1a.el
> EMACS/lisp/textmodes/package-1b.el
> EMACS/lisp/textmodes/package-1c.el
> EMACS/lisp/progmodes/package-2.el
> EMACS/lisp/emacs-lisp/package-3.el

Where 1a, 1b, and 1c are different versions of the same package?  This
is not required; the updates of bundled packages should go into
~/.emacs.d/elpa/, I think.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

>  - Use package.el to initialize and load these during startup

I'm OK with this, as long as it doesn't impact current load-time efficiency.

My past experience with package.is that it introduced a lot of slowness, due
to loading far more autoloads than I might ever care about. This is partly why
use-package manages my load-path and autoloads explicit: I only use maybe 1%
of the total autoloads available in the system.

As long as I'm not paying a cost just for having package.el be part of the
process, I'm fine with this part of your goal.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Eli Zaretskii writes:
> That just means installing a new version needs to remove the previous
> version's files first, that's all.  It has nothing to do with
> load-path.

That's simply not an option when a user wants to install an updated ELPA
package in his home directory while the system administrator keeps the
Emacs installation as-distributed.  In other words, you'd kill _the_
feature that package.el was supposed to give mere users and arguably the
main selling point.

>> Org-mode v1 comes with org-html. v1 is distributed through ELPA.
>> Org-mode v2 comes with ox-html. We upgrade v1 to v2, and v1 is removed
>> from the path. org-html is no longer loadable.
>
> org-html doesn't need to be loadable if it doesn't exist in the new
> version.  Again, nothing to do with load-path.

But in the situation explained by Phillip it _is_ loadable, even though
it must not be.  If in fact you have an autoload pointing to a file that
no longer exists or doesn't contain the autoloaded function anymore,
then a modification of load-path after that autoload has been generated
won't help at all.

>> My conclusion: having org-mode its own directory is a good thing. By
>> extrapolation, therefore, having most or all packages in their own
>> directory would be a good thing.
>
> The extrapolation is invalid, because while Org is a very large
> package, most other packages aren't.

Forget about Org.  The only reason it gets mentioned at all is that it
is moving fast enough to have already produced all the problems
mentioned so far _in the wild_, with actual users suffering the
consequences.  Each such failure can be easily demonstrated with a
package having just a single or two files.

>> > If package.el already knows how to unload the old features and load
>> > the new ones, it will continue doing this for any package, whether in
>> > or out of core.  Right?
>> 
>> With the example that I have given, this fails at the moment for
>> org-mode, specificially because org-mode is in core.
>
> I don't understand why it fails, but if it's a missing feature in
> package.el or in the infrastructure, we need to add that, in order to
> move to having bundled packages on ELPA.

Well, start with cleaning up load-path and autoloads in a way that any
built-in package can be cleanly deactivated: after deactivation Emacs
behaves as if the package wasn't present.

If you think you don't need this, I have done some experiments to excise
the remnants of a built-in package from Emacs' data structures
(autoloads and customization), but there really isn't any supported way
to do that and I consider the whole thing incredibly brittle since I
need to manipulate internal data structures to do that.  At the minimum
you'd need official support for re-defining an autoload so it stops
pointing at the wrong load-file.

Again, the results of those forays are still in (upstream) Org in
various places: testing/org-batch-test-init.el removes enough of the
Emacs' builtin Org package so that the tests actually use the correct
files, there's some more code in mk/org-fixup.el to deal with the
generation of some files that package.el can't produce in its current
state and then some checks in org-version (org.el) to warn about a mixed
installation.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Phillip Lord writes:
>> Paackage.el has it's own limitations which unfortunately are also
>> highlighted by Org.  Some of the files that go into ELPA Org are
>> actually generated by an upstream makefile so that the tarball that ELPA
>> distributes can be successfully installed within those limitations.
>
> Do tell. Which files are generated?

Package.el cannot deal with a two-tiered autoloads structure, so
org-loaddefs.el must be distributed (the first-level org-autoloads is
generated by package.el).  The other generated file is org-version.el.

> Using package.el in core would force us to fix these things, I think,
> which is a good thing!

No other package on ELPA uses such an autoload structure AFAIK.  In
general, package.el or ELPA doesn't expect that anything needs to be
generated (save the .elc files).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Tue, 18 Oct 2016 20:48:29 +0200
> 
> Eli Zaretskii writes:
> > That just means installing a new version needs to remove the previous
> > version's files first, that's all.  It has nothing to do with
> > load-path.
> 
> That's simply not an option when a user wants to install an updated ELPA
> package in his home directory while the system administrator keeps the
> Emacs installation as-distributed.  In other words, you'd kill _the_
> feature that package.el was supposed to give mere users and arguably the
> main selling point.

If no better solution comes up, it would mean the user will have to
restart Emacs.  Hardly a catastrophe.

> >> Org-mode v1 comes with org-html. v1 is distributed through ELPA.
> >> Org-mode v2 comes with ox-html. We upgrade v1 to v2, and v1 is removed
> >> from the path. org-html is no longer loadable.
> >
> > org-html doesn't need to be loadable if it doesn't exist in the new
> > version.  Again, nothing to do with load-path.
> 
> But in the situation explained by Phillip it _is_ loadable, even though
> it must not be.  If in fact you have an autoload pointing to a file that
> no longer exists or doesn't contain the autoloaded function anymore,
> then a modification of load-path after that autoload has been generated
> won't help at all.

Restarting Emacs will solve that.

> Well, start with cleaning up load-path and autoloads in a way that any
> built-in package can be cleanly deactivated: after deactivation Emacs
> behaves as if the package wasn't present.
> 
> If you think you don't need this, I have done some experiments to excise
> the remnants of a built-in package from Emacs' data structures
> (autoloads and customization), but there really isn't any supported way
> to do that and I consider the whole thing incredibly brittle since I
> need to manipulate internal data structures to do that.  At the minimum
> you'd need official support for re-defining an autoload so it stops
> pointing at the wrong load-file.

These problems need to be attacked and solved.  But saying that
instead of solving them we should fundamentally change our lisp/ tree
is IMO not the right way.  It's akin to searching the keys where the
streetlight is, not where they were lost.  We need to define the goal,
and then work towards implementing it, not the other way around.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Achim Gratz <Stromeko@nexgo.de>
>> Date: Tue, 18 Oct 2016 20:48:29 +0200
>> 
>> Eli Zaretskii writes:
>> > That just means installing a new version needs to remove the previous
>> > version's files first, that's all.  It has nothing to do with
>> > load-path.
>> 
>> That's simply not an option when a user wants to install an updated ELPA
>> package in his home directory while the system administrator keeps the
>> Emacs installation as-distributed.  In other words, you'd kill _the_
>> feature that package.el was supposed to give mere users and arguably the
>> main selling point.
>
> If no better solution comes up, it would mean the user will have to
> restart Emacs.  Hardly a catastrophe.

Eli, I think you are not taking the time here to understand the problem.
Both Achim and I have thought the issue through, and it restarting Emacs
were the solution then neither of us would, I expect, be complaining.

As Achim says, an org-html in the core installation cannot be removed
because it comes as part of a packaged Emacs and needs admin
privilleges; again, can be done, but not very good practice. The
org-html in core can be shadowed but only by another file called
org-html. More recent versions of org do not contain such a file.

There are solutions to this, many of them hacky. The simplest solution
is to NOT include the directory containing the old version of org in the
load-path.

Currently, package.el does not do this for core installed files, but
does do this for package.el installed files. And, package.el could
*only* be made to do this, if a package is in its own directory.


>> But in the situation explained by Phillip it _is_ loadable, even though
>> it must not be.  If in fact you have an autoload pointing to a file that
>> no longer exists or doesn't contain the autoloaded function anymore,
>> then a modification of load-path after that autoload has been generated
>> won't help at all.
>
> Restarting Emacs will solve that.

It will not, and does not.

>> If you think you don't need this, I have done some experiments to excise
>> the remnants of a built-in package from Emacs' data structures
>> (autoloads and customization), but there really isn't any supported way
>> to do that and I consider the whole thing incredibly brittle since I
>> need to manipulate internal data structures to do that.  At the minimum
>> you'd need official support for re-defining an autoload so it stops
>> pointing at the wrong load-file.
>
> These problems need to be attacked and solved.  But saying that
> instead of solving them we should fundamentally change our lisp/ tree
> is IMO not the right way.  It's akin to searching the keys where the
> streetlight is, not where they were lost.  We need to define the goal,
> and then work towards implementing it, not the other way around.

I have defined my goal and a path toward it.

I believe that Emacs should have a single mechanism for managing
packages, and that this mechanism is package.el. I am aware that this is
a big change and have, therefore, presented an incremental path by which
this can be trialed out without mass renaming or moves of files in core,
and which also allows support for adding packages stored in ELPA to the
core build which is a sensible goal put forward by John.

I will leave it there. If neither you or John want to go this way, I
will stop.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> I could less easily do:
>> 
>> EMACS/lisp
>> EMACS/lisp/lots-of-files.el
>> EMACS/lisp/textmodes/package-1
>> EMACS/lisp/progmodes/package-2
>> EMACS/lisp/emacs-lisp/package-3
>
> This is what we want.  IOW, we want foo.el to live in the same
> directory it is now, even if its Git repository will be on ELPA.

I have been unclear. package-1, package-2 here are directorires.


>> This requires updating the current build process to exclude multiple
>> directories
>
> Why do we need to exclude directories during the build?  And what does
> "exclude" mean in this context?

Like "obsolete". Not compile them or include them in loaddefs.el gneration.


>> I also dislike this, because I think it mixes up files of different
>> kinds in the tree (that is files which are in place in the Emacs git
>> repo, and those which are not).
>
> There are no different kinds.

Files stored in emacs git and elpa git are different kinds of file,
because they will require diffferent proceedures to update.

They are all of the same kind: Lisp
> packages that are bundled with a release tarball.  The fact that some
> of them are in a separate Git repo  doesn't make them a different
> kind.  We want the move to ELPA to be transparent as far as the Emacs
> lisp/ directory structure is concerned.
>
>> Finally, it would no doubt be possible to do:
>> 
>> EMACS/lisp
>> EMACS/lisp/textmodes/package-1a.el
>> EMACS/lisp/textmodes/package-1b.el
>> EMACS/lisp/textmodes/package-1c.el
>> EMACS/lisp/progmodes/package-2.el
>> EMACS/lisp/emacs-lisp/package-3.el
>
> Where 1a, 1b, and 1c are different versions of the same package?

No, where 1a, 1b and 1c are different modules of the same package, such
as org.el, org-export, org-cal etc.

> This is not required; the updates of bundled packages should go into
> ~/.emacs.d/elpa/, I think.

That's the only place that they can go, of course.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>  - Use package.el to initialize and load these during startup
>
> I'm OK with this, as long as it doesn't impact current load-time efficiency.
>
> My past experience with package.is that it introduced a lot of slowness, due
> to loading far more autoloads than I might ever care about. This is partly why
> use-package manages my load-path and autoloads explicit: I only use maybe 1%
> of the total autoloads available in the system.
>
> As long as I'm not paying a cost just for having package.el be part of the
> process, I'm fine with this part of your goal.

I have no a priori way of determining whether this will be the case or
not. We can only find this out by building it.

It might even make things quicker -- currently Emacs loads all the
autoloads in loaddefs.el whether you want it to or not. We might be able
to reduce that. My own feeling is that autoloads are currently used for
two things: ensuring that a user feature is available (i.e. open a perl
file and get perl mode); and enabling a programmatic feature without
needing a "require" form (i.e. in a new lisp file call "pcase").

What I would say, though, is that IF it does slow things down, then this
will provide the impetus to speed package.el up, to the benefit of
everyone not using use-package.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Achim Gratz <Stromeko@nexgo.de> writes:

> Phillip Lord writes:
>>> Paackage.el has it's own limitations which unfortunately are also
>>> highlighted by Org.  Some of the files that go into ELPA Org are
>>> actually generated by an upstream makefile so that the tarball that ELPA
>>> distributes can be successfully installed within those limitations.
>>
>> Do tell. Which files are generated?
>
> Package.el cannot deal with a two-tiered autoloads structure, so
> org-loaddefs.el must be distributed (the first-level org-autoloads is
> generated by package.el).  The other generated file is org-version.el.
>
>> Using package.el in core would force us to fix these things, I think,
>> which is a good thing!
>
> No other package on ELPA uses such an autoload structure AFAIK.  In
> general, package.el or ELPA doesn't expect that anything needs to be
> generated (save the .elc files).


Yeah, that's true, and it's a pain. Several of my own packages generate
their own documentation, and there is no way to support this. Likewise,
the inability to support texinfo files and compile these.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Achim Gratz <Stromeko@nexgo.de>,  emacs-devel@gnu.org
> Date: Wed, 19 Oct 2016 08:51:39 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Achim Gratz <Stromeko@nexgo.de>
> >> Date: Tue, 18 Oct 2016 20:48:29 +0200
> >> 
> >> Eli Zaretskii writes:
> >> > That just means installing a new version needs to remove the previous
> >> > version's files first, that's all.  It has nothing to do with
> >> > load-path.
> >> 
> >> That's simply not an option when a user wants to install an updated ELPA
> >> package in his home directory while the system administrator keeps the
> >> Emacs installation as-distributed.  In other words, you'd kill _the_
> >> feature that package.el was supposed to give mere users and arguably the
> >> main selling point.
> >
> > If no better solution comes up, it would mean the user will have to
> > restart Emacs.  Hardly a catastrophe.
> 
> Eli, I think you are not taking the time here to understand the problem.
> Both Achim and I have thought the issue through

Those kinds of remarks are hardly constructive.  Please assume that
I've thought the issues through as best I could, and if I'm missing
some important details, those details should be put on the table
explicitly and discussed.

> and it restarting Emacs were the solution then neither of us would,
> I expect, be complaining.

I don't see why restarting won't be a solution.  If load-path was
re-arranged to put the latest version of a package first, and the
package's autoloads are on a file that has been regenerated, what else
is missing to make restart the correct solution?

> As Achim says, an org-html in the core installation cannot be removed
> because it comes as part of a packaged Emacs and needs admin
> privilleges; again, can be done, but not very good practice. The
> org-html in core can be shadowed but only by another file called
> org-html. More recent versions of org do not contain such a file.

If load-path is rearranged when a newer version of Org is installed,
and org-loaddefs are regenerated, then, after a restart, any 'load'
and 'require' will find the new files first, and the old ones will be
effectively invisible to Emacs.  Right?

> There are solutions to this, many of them hacky. The simplest solution
> is to NOT include the directory containing the old version of org in the
> load-path.

That solution will only work for packages in their own directories.
We want to have a solution that works even for files in common
directories.  I think rearranging load-path and regenerating the
autoloads will solve that as well.

> >> But in the situation explained by Phillip it _is_ loadable, even though
> >> it must not be.  If in fact you have an autoload pointing to a file that
> >> no longer exists or doesn't contain the autoloaded function anymore,
> >> then a modification of load-path after that autoload has been generated
> >> won't help at all.
> >
> > Restarting Emacs will solve that.
> 
> It will not, and does not.

Why won't it?  (The "does not" part just means package.el doesn't
currently do that, AFAIU, but it can be extended to do it.)

> I have defined my goal and a path toward it.
> 
> I believe that Emacs should have a single mechanism for managing
> packages, and that this mechanism is package.el. I am aware that this is
> a big change and have, therefore, presented an incremental path by which
> this can be trialed out without mass renaming or moves of files in core,
> and which also allows support for adding packages stored in ELPA to the
> core build which is a sensible goal put forward by John.
> 
> I will leave it there. If neither you or John want to go this way, I
> will stop.

I think we both indicated that we want to go this way, we just don't
want the Lisp files scattered among dozens of directories, each one
with a single file.  Yes, this is a goal that is harder to achieve,
but I don't yet see any fundamental difficulties on that path, just
some more work.

I think making such a change incrementally is worse than doing it at
once, as far as the directory structure is concerned.  We don't want
to change the directory structure several times, ideally not even
once.  But if some change is required, it should be done in one go, so
we need to decide on the structure up front.

Thanks.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

> If load-path is rearranged when a newer version of Org is installed,
> and org-loaddefs are regenerated, then, after a restart, any 'load'
> and 'require' will find the new files first, and the old ones will be
> effectively invisible to Emacs.  Right?

I think that regenerating *-loaddefs as Eli pointed out will solve the
effective issue at hand.  What's not covered though is that it might
still be possible to explicitly load/require an old file that is no
longer there in the new version - e.g. (load "org-html") using the
example given by Phillip.  But will that really be a problem at all?
And if it is, we may find a solution for this.  And we have to keep in
mind that after all, this will most probably only happen with large
packages that will have their own directory anyway.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> CC: Phillip Lord <phillip.lord@russet.org.uk>, <Stromeko@nexgo.de>,
> 	<emacs-devel@gnu.org>
> Date: Wed, 19 Oct 2016 11:31:11 +0200
> 
> I think that regenerating *-loaddefs as Eli pointed out will solve the
> effective issue at hand.  What's not covered though is that it might
> still be possible to explicitly load/require an old file that is no
> longer there in the new version - e.g. (load "org-html") using the
> example given by Phillip.

The new version of Org will never do that.  So you are talking about
other packages that were not yet updated to follow suit, is that
right?  If so, is such explicit loading allowed?  If it's allowed,
it's a separate problem of dependencies between packages, and should
exist with any arrangement of directories, I think.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alain Schneble <a.s@realize.ch>
>> CC: Phillip Lord <phillip.lord@russet.org.uk>, <Stromeko@nexgo.de>,
>> 	<emacs-devel@gnu.org>
>> Date: Wed, 19 Oct 2016 11:31:11 +0200
>> 
>> I think that regenerating *-loaddefs as Eli pointed out will solve the
>> effective issue at hand.  What's not covered though is that it might
>> still be possible to explicitly load/require an old file that is no
>> longer there in the new version - e.g. (load "org-html") using the
>> example given by Phillip.
>
> The new version of Org will never do that.  So you are talking about
> other packages that were not yet updated to follow suit, is that
> right?  If so, is such explicit loading allowed?  If it's allowed,
> it's a separate problem of dependencies between packages, and should
> exist with any arrangement of directories, I think.

Exactly, I see it the same way.  I also think it's as a separate issue
(that after all could also be handled by each package separately if it
needs to solve it at all).

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Alain Schneble <a.s@realize.ch> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Alain Schneble <a.s@realize.ch>
>>> CC: Phillip Lord <phillip.lord@russet.org.uk>, <Stromeko@nexgo.de>,
>>> 	<emacs-devel@gnu.org>
>>> Date: Wed, 19 Oct 2016 11:31:11 +0200
>>> 
>>> I think that regenerating *-loaddefs as Eli pointed out will solve the
>>> effective issue at hand.  What's not covered though is that it might
>>> still be possible to explicitly load/require an old file that is no
>>> longer there in the new version - e.g. (load "org-html") using the
>>> example given by Phillip.
>>
>> The new version of Org will never do that.  So you are talking about
>> other packages that were not yet updated to follow suit, is that
>> right?  If so, is such explicit loading allowed?  If it's allowed,
>> it's a separate problem of dependencies between packages, and should
>> exist with any arrangement of directories, I think.
>
> Exactly, I see it the same way.  I also think it's as a separate issue
> (that after all could also be handled by each package separately if it
> needs to solve it at all).

To elaborate what I wrote in parenthesis in the last sentence above: In
this situation, the new package could still (provide
'the-deleted-feature) or register a function to be called when loading
such deprecated file using after-load-alist or somesuch.  The former
could make a (require 'the-deleted-feature) a "noop", the latter could
raise an error, or whatever.  And there are for sure other feasible
approaches that could be implemented by a package if required.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Eli, I think you are not taking the time here to understand the problem.
>> Both Achim and I have thought the issue through
>
> Those kinds of remarks are hardly constructive.  Please assume that
> I've thought the issues through as best I could,

If you will assume the same, yes, of course.

>> and it restarting Emacs were the solution then neither of us would,
>> I expect, be complaining.
>
> I don't see why restarting won't be a solution.  If load-path was
> re-arranged to put the latest version of a package first, and the
> package's autoloads are on a file that has been regenerated, what else
> is missing to make restart the correct solution?


Again, only if the file names have not changed between one release and
the next. This is an assumption which is not true. And, anyway, it's not
possible to regenerate autoloads in core emacs.

>> As Achim says, an org-html in the core installation cannot be removed
>> because it comes as part of a packaged Emacs and needs admin
>> privilleges; again, can be done, but not very good practice. The
>> org-html in core can be shadowed but only by another file called
>> org-html. More recent versions of org do not contain such a file.
>
> If load-path is rearranged when a newer version of Org is installed,
> and org-loaddefs are regenerated, then, after a restart, any 'load'
> and 'require' will find the new files first, and the old ones will be
> effectively invisible to Emacs.  Right?

No. They are still there, and are still visible, as long as they are in
the load path.


>> There are solutions to this, many of them hacky. The simplest solution
>> is to NOT include the directory containing the old version of org in the
>> load-path.
>
> That solution will only work for packages in their own directories.

Yes, hence my assumption my argument that packages in their own
directories is a good thing.

> We want to have a solution that works even for files in common
> directories.  I think rearranging load-path and regenerating the
> autoloads will solve that as well.

As discussed rearranging the load path does not solve the problem, and
regenerating the autoloads isn't possible.


>> It will not, and does not.
>
> Why won't it?  (The "does not" part just means package.el doesn't
> currently do that, AFAIU, but it can be extended to do it.)

load-path is directory based. I cannot prevent files that could be
loaded from core, unless they are in one directory.


>> I will leave it there. If neither you or John want to go this way, I
>> will stop.
>
> I think we both indicated that we want to go this way, we just don't
> want the Lisp files scattered among dozens of directories, each one
> with a single file.  Yes, this is a goal that is harder to achieve,
> but I don't yet see any fundamental difficulties on that path, just
> some more work.

The fundamental difficult is that we are supporting two different file
arrangements for any package present in core that can also be installed
from ELPA.


> I think making such a change incrementally is worse than doing it at
> once, as far as the directory structure is concerned.  We don't want
> to change the directory structure several times, ideally not even
> once.  But if some change is required, it should be done in one go, so
> we need to decide on the structure up front.

That, of course, is also a possibility, and one that I would be happy to
consider.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alain Schneble <a.s@realize.ch>
>> CC: Phillip Lord <phillip.lord@russet.org.uk>, <Stromeko@nexgo.de>,
>> 	<emacs-devel@gnu.org>
>> Date: Wed, 19 Oct 2016 11:31:11 +0200
>> 
>> I think that regenerating *-loaddefs as Eli pointed out will solve the
>> effective issue at hand.  What's not covered though is that it might
>> still be possible to explicitly load/require an old file that is no
>> longer there in the new version - e.g. (load "org-html") using the
>> example given by Phillip.
>
> The new version of Org will never do that.  So you are talking about
> other packages that were not yet updated to follow suit, is that
> right?  If so, is such explicit loading allowed?  If it's allowed,
> it's a separate problem of dependencies between packages, and should
> exist with any arrangement of directories, I think.


org has many add on packages, which do use explicit "require" forms to
internal packages (if, for example, they extend an existing org
backend).

What I would expect is that an explicit (require 'org-html) would fail
(i.e. report an error) when upgrading org to a version that does not
include org-html. What actually happens is org-html from the old version
gets loaded.

load-path shadowing is I think, an ineffective mechanism for isolation
between versions. Deleting old versions would work but not for core,
admin installed, packages and is not-revertable. Removing old versions
from load-path is clean and the best solution, which is why package.el
uses is.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> Alain Schneble <a.s@realize.ch> writes:
>> Exactly, I see it the same way.  I also think it's as a separate issue
>> (that after all could also be handled by each package separately if it
>> needs to solve it at all).
>
> To elaborate what I wrote in parenthesis in the last sentence above: In
> this situation, the new package could still (provide
> 'the-deleted-feature) or register a function to be called when loading
> such deprecated file using after-load-alist or somesuch.  The former
> could make a (require 'the-deleted-feature) a "noop", the latter could
> raise an error, or whatever.  And there are for sure other feasible
> approaches that could be implemented by a package if required.


Just to clear, you are suggesting that when moving from org-html to
ox-html, the org developers should have had a file called org-html.el
like so:

(error "org-html is now deprecated")
(provide 'org-html)


Yes, that would work, but does not provide a good solution to my mind.

I really must stop this. We are going around in circles. I will write
this discussion up, and add it to the README on my branch.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Wed, 19 Oct 2016 16:09:25 +0100
> 
> >> and it restarting Emacs were the solution then neither of us would,
> >> I expect, be complaining.
> >
> > I don't see why restarting won't be a solution.  If load-path was
> > re-arranged to put the latest version of a package first, and the
> > package's autoloads are on a file that has been regenerated, what else
> > is missing to make restart the correct solution?
> 
> Again, only if the file names have not changed between one release and
> the next.

Sorry, I don't see why changes in file names matter.  Please
elaborate.

If the old file org-html.el exists in directory DIR1, and the new file
ox-html.el that defines the same features lives in directory DIR2,
then having DIR2 in load-path ahead of DIR1 will always find
ox-html.el before it finds org-html.el.  If the autoloads for symbols
previously defined in org-html.el now point to ox-html.el, referencing
such a symbol will load ox-html.el because the regenerated autoloads
say so.  Right?

> And, anyway, it's not possible to regenerate autoloads in core
> emacs.

Why not?  We do that all the time, each time we rebuild Emacs.

> > If load-path is rearranged when a newer version of Org is installed,
> > and org-loaddefs are regenerated, then, after a restart, any 'load'
> > and 'require' will find the new files first, and the old ones will be
> > effectively invisible to Emacs.  Right?
> 
> No. They are still there, and are still visible, as long as they are in
> the load path.

Why do we care that the files are there if no one and nothing is
loading them?

Files are loaded via calls to 'load' or 'require', and implicitly by
calling or referencing an autoloaded symbol.  The former is taken care
of by load-path: the directory with newer files will be searched ahead
of the directory with older ones.  The latter is taken care of by
regenerating autoloads, because the autoload cookie for a symbol will
now mention either the file of the same name, in which case it will be
loaded from the newer directory (because of load-path reordering), or
it will mention the file of a newer name which defines the symbol, and
which don't exist at all in the old directory.

What situation will not be caught by these two devices?

> load-path is directory based. I cannot prevent files that could be
> loaded from core, unless they are in one directory.

You don't need to prevent that.  Files that live only in the core
directories will not be affected by having
.emacs.d/packages/elpa/something earlier in load-path, because no file
by the same name will be found in the core directories.

> >> I will leave it there. If neither you or John want to go this way, I
> >> will stop.
> >
> > I think we both indicated that we want to go this way, we just don't
> > want the Lisp files scattered among dozens of directories, each one
> > with a single file.  Yes, this is a goal that is harder to achieve,
> > but I don't yet see any fundamental difficulties on that path, just
> > some more work.
> 
> The fundamental difficult is that we are supporting two different file
> arrangements for any package present in core that can also be installed
> from ELPA.

It isn't fundamental, in the sense that I used this word, because it
can be solved quite easily.  "Fundamental" in this case means
something requires a thorough redesign of many Emacs basic features.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
>> Date: Wed, 19 Oct 2016 16:09:25 +0100
>> 
>> >> and it restarting Emacs were the solution then neither of us would,
>> >> I expect, be complaining.
>> >
>> > I don't see why restarting won't be a solution.  If load-path was
>> > re-arranged to put the latest version of a package first, and the
>> > package's autoloads are on a file that has been regenerated, what else
>> > is missing to make restart the correct solution?
>> 
>> Again, only if the file names have not changed between one release and
>> the next.

> If the old file org-html.el exists in directory DIR1, and the new file
> ox-html.el that defines the same features lives in directory DIR2,
> then having DIR2 in load-path ahead of DIR1 will always find
> ox-html.el before it finds org-html.el.  If the autoloads for symbols
> previously defined in org-html.el now point to ox-html.el, referencing
> such a symbol will load ox-html.el because the regenerated autoloads
> say so.  Right?

If everybody does the right thing. If not everybody does the right
thing, the old version gets loaded. Happened to me, it's hard to debug
and hard to work out what is going wrong.


>> And, anyway, it's not possible to regenerate autoloads in core
>> emacs.
>
> Why not?  We do that all the time, each time we rebuild Emacs.

Developers do this all the time, users do not.

We could remove, of course, put autoloads in user space under the
control of the user. This is what package.el does.

>> > If load-path is rearranged when a newer version of Org is installed,
>> > and org-loaddefs are regenerated, then, after a restart, any 'load'
>> > and 'require' will find the new files first, and the old ones will be
>> > effectively invisible to Emacs.  Right?
>> 
>> No. They are still there, and are still visible, as long as they are in
>> the load path.
>
> Why do we care that the files are there if no one and nothing is
> loading them?

They are loadable. They should not be. It's hit me, and it's a pain.

> What situation will not be caught by these two devices?

Described this several times.

>> load-path is directory based. I cannot prevent files that could be
>> loaded from core, unless they are in one directory.
>
> You don't need to prevent that.  Files that live only in the core
> directories will not be affected by having
> .emacs.d/packages/elpa/something earlier in load-path, because no file
> by the same name will be found in the core directories.

Also, described this several times.


>> >> I will leave it there. If neither you or John want to go this way, I
>> >> will stop.
>> >
>> > I think we both indicated that we want to go this way, we just don't
>> > want the Lisp files scattered among dozens of directories, each one
>> > with a single file.  Yes, this is a goal that is harder to achieve,
>> > but I don't yet see any fundamental difficulties on that path, just
>> > some more work.
>> 
>> The fundamental difficult is that we are supporting two different file
>> arrangements for any package present in core that can also be installed
>> from ELPA.
>
> It isn't fundamental, in the sense that I used this word, because it
> can be solved quite easily.  "Fundamental" in this case means
> something requires a thorough redesign of many Emacs basic features.

Okay, that's fine. I don't see any fundamental difficulties in following
a package.el based solution either; so far, the only issue that I have
seen is "I like the directory structure the way it is", and "I don't
want to see lots of directories with one or two files in core, although
it's fine in user space".

Seriously, honestly, definately, I will stop defending my proposal now,
because the discussion is not advancing at all.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Ted Zlatanov]

On Wed, 19 Oct 2016 17:12:30 +0100 phillip.lord@russet.org.uk (Phillip Lord) wrote: 

PL> Okay, that's fine. I don't see any fundamental difficulties in following
PL> a package.el based solution either; so far, the only issue that I have
PL> seen is "I like the directory structure the way it is", and "I don't
PL> want to see lots of directories with one or two files in core, although
PL> it's fine in user space".

I'd rather have the "lots of directories" setup, if I had to choose.
It's significantly better in the long term IMHO, so it's worth
short-term inconvenience and acclimation.

Ted

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Alain Schneble <a.s@realize.ch>,  Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Wed, 19 Oct 2016 16:14:44 +0100
> 
> > The new version of Org will never do that.  So you are talking about
> > other packages that were not yet updated to follow suit, is that
> > right?  If so, is such explicit loading allowed?  If it's allowed,
> > it's a separate problem of dependencies between packages, and should
> > exist with any arrangement of directories, I think.
> 
> org has many add on packages, which do use explicit "require" forms to
> internal packages (if, for example, they extend an existing org
> backend).

Please see above: the problem is not within a single package, because
any package will always maintain internal requires in order.

> What I would expect is that an explicit (require 'org-html) would fail
> (i.e. report an error) when upgrading org to a version that does not
> include org-html. What actually happens is org-html from the old version
> gets loaded.

There should be no (require 'org-html) inside Org files in an Org
version that doesn't include org-html.

> load-path shadowing is I think, an ineffective mechanism for isolation
> between versions. Deleting old versions would work but not for core,
> admin installed, packages and is not-revertable.

Which is why load-path shadowing is the mechanism we should use, all
its disadvantages notwithstanding.

> Removing old versions from load-path is clean and the best solution,
> which is why package.el uses is.

That solution won't work with files in core.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Wed, 19 Oct 2016 17:12:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If the old file org-html.el exists in directory DIR1, and the new file
> > ox-html.el that defines the same features lives in directory DIR2,
> > then having DIR2 in load-path ahead of DIR1 will always find
> > ox-html.el before it finds org-html.el.  If the autoloads for symbols
> > previously defined in org-html.el now point to ox-html.el, referencing
> > such a symbol will load ox-html.el because the regenerated autoloads
> > say so.  Right?
> 
> If everybody does the right thing. If not everybody does the right
> thing, the old version gets loaded. Happened to me, it's hard to debug
> and hard to work out what is going wrong.

OK, so we agree that if everybody does the right thing, this will
work.  That's all we need at this point, because we certainly don't
need to cater to packages which do the wrong thing: such packages will
be fixed in no time, because they are bundled and are part of any
Emacs installation.

> >> And, anyway, it's not possible to regenerate autoloads in core
> >> emacs.
> >
> > Why not?  We do that all the time, each time we rebuild Emacs.
> 
> Developers do this all the time, users do not.
> 
> We could remove, of course, put autoloads in user space under the
> control of the user. This is what package.el does.

Exactly.  So we should use what package.el does, only extend it so it
does that correctly for files under lisp/ that come from ELPA.

> >> > If load-path is rearranged when a newer version of Org is installed,
> >> > and org-loaddefs are regenerated, then, after a restart, any 'load'
> >> > and 'require' will find the new files first, and the old ones will be
> >> > effectively invisible to Emacs.  Right?
> >> 
> >> No. They are still there, and are still visible, as long as they are in
> >> the load path.
> >
> > Why do we care that the files are there if no one and nothing is
> > loading them?
> 
> They are loadable.

They are, but if no one is loading them, they don't matter.

> Seriously, honestly, definately, I will stop defending my proposal now,
> because the discussion is not advancing at all.

I hope we've reached the agreement that the lisp/ tree as it is can
continue be used when some packages in it come from ELPA.

Thanks.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Eli Zaretskii writes:
> I don't see why restarting won't be a solution.  If load-path was
> re-arranged to put the latest version of a package first, and the
> package's autoloads are on a file that has been regenerated, what else
> is missing to make restart the correct solution?

You are missing that the order in which all these things happen becomes
important.  The newer autoload can only shadow the old one if it's
processed _before_, likewise initializing other Emacs packages might
inadvertently pull in old autoloads or definitions before things have
been rearranged by package.el.  This is not hypothetical, all of that
has already happened and users have suffered from it.  When they report
the resulting symptoms it is extremely difficult to tease out what
exactly happened, in which order and why, so it also wastes the time of
those folks who want to help them.

> If load-path is rearranged when a newer version of Org is installed,
> and org-loaddefs are regenerated, then, after a restart, any 'load'
> and 'require' will find the new files first, and the old ones will be
> effectively invisible to Emacs.  Right?

If you only want to look at that one-dimensional example, yes.  Real
Emacs installations are quite a bit more varied.  The tools we have
today to sove this problem make things more brittle when they should
become more robust.

> That solution will only work for packages in their own directories.
> We want to have a solution that works even for files in common
> directories.  I think rearranging load-path and regenerating the
> autoloads will solve that as well.

I still don't get what you hope to gain from throwing things that don't
belong together (and aren't upstrem) in a single directory (only talking
about the code parts here, documentation can be pulled someplace else).

In fact, most of the packages that might be on ELPA already are in their
own subdirectories and the only thing you'll need to do is not generate
the first-level autoloads into loaddefs.el and instead have some
package-autoloads be generated by package.el.  Then create some default
package configuration with all these packages activated, but give the
user configuration preference so they can be deactivated.

> I think we both indicated that we want to go this way, we just don't
> want the Lisp files scattered among dozens of directories, each one
> with a single file.  Yes, this is a goal that is harder to achieve,
> but I don't yet see any fundamental difficulties on that path, just
> some more work.

Single-file packages in core that are duplicated on ELPA (if these even
exist, I haven't checked) can coexist just fine in a single directory if
you so desire.  They are orders of magnitude less complicated to deal
with.  But the total of these should still live in a separate package
directory that is created just to give them a home.

> I think making such a change incrementally is worse than doing it at
> once, as far as the directory structure is concerned.  We don't want
> to change the directory structure several times, ideally not even
> once.  But if some change is required, it should be done in one go, so
> we need to decide on the structure up front.

I've said it before and I will say it again just this one time: If Emacs
takes packetization of the core seriously, then the "hard core" should
contain just the stuff that is needed for bootstrapping Emacs.
Everything else should be a package.  Single-file packages would live in
one directory together, and multi-file packages would be in their own
directory each.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eli Zaretskii writes:
>> I don't see why restarting won't be a solution.  If load-path was
>> re-arranged to put the latest version of a package first, and the
>> package's autoloads are on a file that has been regenerated, what else
>> is missing to make restart the correct solution?
>
> You are missing that the order in which all these things happen becomes
> important.  The newer autoload can only shadow the old one if it's
> processed _before_, likewise initializing other Emacs packages might
> inadvertently pull in old autoloads or definitions before things have
> been rearranged by package.el.  This is not hypothetical, all of that
> has already happened and users have suffered from it.  When they report
> the resulting symptoms it is extremely difficult to tease out what
> exactly happened, in which order and why, so it also wastes the time of
> those folks who want to help them.

Of course.  The order of directories in load path is always relevant.
Whatever structure we choose.

>> I think making such a change incrementally is worse than doing it at
>> once, as far as the directory structure is concerned.  We don't want
>> to change the directory structure several times, ideally not even
>> once.  But if some change is required, it should be done in one go, so
>> we need to decide on the structure up front.
>
> I've said it before and I will say it again just this one time: If Emacs
> takes packetization of the core seriously, then the "hard core" should
> contain just the stuff that is needed for bootstrapping Emacs.

That has nothing to do with the directory layout.  This is possible if
we keep the ./lisp directory layout.

> Everything else should be a package.  Single-file packages would live in
> one directory together, and multi-file packages would be in their own
> directory each.

That will be more or less what will be mostly the case if we keep the
./lisp directory layout.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Stefan Monnier]

> I'd rather have the "lots of directories" setup, if I had to choose.
> It's significantly better in the long term IMHO, so it's worth
> short-term inconvenience and acclimation.

If we take the files bundled with Emacs and look at how many ELPA
packages they would turn into, we'd end up with a load path that has
more than 200 elements.

FWIW, I've always been wary of having "one dir per package" because of
the performance impact of having a thousand directories in load-path.
Lookup in a large directory is easily optimized to O(log N) or something
like that, but a search along load-path won't be similarly optimized by
the kernel.

[ In `install.el` (an old package that predates package.el), I installed
  all single-file packages into a shared directory specifically for
  that reason.  ]

This is not based on actual experience, tho, so maybe I'm worrying about
nothing.


        Stefan

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>
>> Alain Schneble <a.s@realize.ch> writes:
>>> Exactly, I see it the same way.  I also think it's as a separate issue
>>> (that after all could also be handled by each package separately if it
>>> needs to solve it at all).
>>
>> To elaborate what I wrote in parenthesis in the last sentence above: In
>> this situation, the new package could still (provide
>> 'the-deleted-feature) or register a function to be called when loading
>> such deprecated file using after-load-alist or somesuch.  The former
>> could make a (require 'the-deleted-feature) a "noop", the latter could
>> raise an error, or whatever.  And there are for sure other feasible
>> approaches that could be implemented by a package if required.
>
>
> Just to clear, you are suggesting that when moving from org-html to
> ox-html, the org developers should have had a file called org-html.el
> like so:

Let me back up first before answering this.  I fully agree with what Eli
said, that if there is a problem, it's not within a single package that
got updated.  Because the updated package won't load a removed file or
if so, then it is simply a bug.

Now if there is another package or some "loose fragment" of code that is
requiring/loading the old feature/file that is no longer present in the
new version, then it's really an inter-package dependency problem.  And
you will always be able to construct cases that won't work properly.
Regardless of the chosen directory structure.

The point I'm trying to make is that if a given package wants to account
for such "misuses" explicitly, it is free to do so.  And there are
feasible solutions to this that a package can choose to provide if it
wants.

> (error "org-html is now deprecated")
> (provide 'org-html)
>
>
> Yes, that would work, but does not provide a good solution to my mind.

Yes, that is one approach.  The other I proposed is using
after-load-alist:

Extract the example in the attached file.

[-- Attachment #2: deprecated-feature-example --]
[-- Type: application/octet-stream, Size: 457 bytes --]

[-- Attachment #3: Type: text/plain, Size: 593 bytes --]


- emacs -Q

- Eval the following fragment, after having replaced [path-to] to the
  proper path where the example was extracted to:

  (push "[path-to]/deprecated-feature-example/foo-package-1.0" load-path)
  (push "[path-to]/deprecated-feature-example/foo-package-2.0" load-path)
  (require 'foo-old)
  => an error will be signaled "foo-old is no longer supported..."

But don't get me wrong.  I really don't think this will be the usual
case.  In most if not all packages, accounting for such misuses won't be
required at all.  I just wanted to prove it's not an unsolvable problem.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Alain Schneble writes:
> Of course.  The order of directories in load path is always relevant.
> Whatever structure we choose.

Good.  Now take that last step: if you want to be sure that you don't
load something you don't want to be loaded, it must not be found via
load-path at all times during the lifetime of the Emacs process.  That
precludes almost all solutions that depend on removal or shadowing
during later steps in the initialization sequence and where
not-yet-to-be-initialized packages become visible when initializing
another package.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Alain Schneble <a.s@realize.ch> writes:

> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Eli Zaretskii writes:
>>> I don't see why restarting won't be a solution.  If load-path was
>>> re-arranged to put the latest version of a package first, and the
>>> package's autoloads are on a file that has been regenerated, what else
>>> is missing to make restart the correct solution?
>>
>> You are missing that the order in which all these things happen becomes
>> important.  The newer autoload can only shadow the old one if it's
>> processed _before_, likewise initializing other Emacs packages might
>> inadvertently pull in old autoloads or definitions before things have
>> been rearranged by package.el.  This is not hypothetical, all of that
>> has already happened and users have suffered from it.  When they report
>> the resulting symptoms it is extremely difficult to tease out what
>> exactly happened, in which order and why, so it also wastes the time of
>> those folks who want to help them.
>
> Of course.  The order of directories in load path is always relevant.
> Whatever structure we choose.

I just realized you were talking about autoloads and not load path.
Sorry.  But the conclusion doesn't change.  We just have to make sure
that the old autoload definitions are no longer loaded.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> If everybody does the right thing. If not everybody does the right
>> thing, the old version gets loaded. Happened to me, it's hard to debug
>> and hard to work out what is going wrong.
>
> OK, so we agree that if everybody does the right thing, this will
> work.  That's all we need at this point, because we certainly don't
> need to cater to packages which do the wrong thing: such packages will
> be fixed in no time, because they are bundled and are part of any
> Emacs installation.

We should cater to packages which do the wrong thing, by ensuring that
the break quickly and obviously.


>> Seriously, honestly, definately, I will stop defending my proposal now,
>> because the discussion is not advancing at all.
>
> I hope we've reached the agreement that the lisp/ tree as it is can
> continue be used when some packages in it come from ELPA.

No, we've agreed to disagree. I think my solution is better, but I also
think that you have more say in these matters than I. I have no desire
to flog a dead horse, which is why I will withdraw the suggestion.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'd rather have the "lots of directories" setup, if I had to choose.
>> It's significantly better in the long term IMHO, so it's worth
>> short-term inconvenience and acclimation.
>
> If we take the files bundled with Emacs and look at how many ELPA
> packages they would turn into, we'd end up with a load path that has
> more than 200 elements.

My load-path has 151 at the moment. Although a lot of these appear to be
duplicates for some reason.


> FWIW, I've always been wary of having "one dir per package" because of
> the performance impact of having a thousand directories in load-path.
> Lookup in a large directory is easily optimized to O(log N) or something
> like that, but a search along load-path won't be similarly optimized by
> the kernel.
>
> [ In `install.el` (an old package that predates package.el), I installed
>   all single-file packages into a shared directory specifically for
>   that reason.  ]
> This is not based on actual experience, tho, so maybe I'm worrying about
> nothing.

I think it's a valid concern, although as with all of these things, it's
hard to tell without building it and trying it. I guess you would mostly
be worried about start up?

Of course, with a few heuristics, it would be possible to speed this up.
For example, if we had a "one package per directory" layout, we could
have `load` look for `blah.el` or `blah-subpackage.el` in the "blah"
directory first. If package.el stored the current version of each
package, somewhere, you could make this lookup pretty close to constant
time.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Achim Gratz <Stromeko@nexgo.de> writes:

> Alain Schneble writes:
>> Of course.  The order of directories in load path is always relevant.
>> Whatever structure we choose.
>
> Good.  Now take that last step: if you want to be sure that you don't
> load something you don't want to be loaded, it must not be found via
> load-path at all times during the lifetime of the Emacs process.

Sorry I can't follow.  The last step of what?  Who wants to load
something we don't want to be loaded?

>                                                                   That
> precludes almost all solutions that depend on removal or shadowing
> during later steps in the initialization sequence and where
> not-yet-to-be-initialized packages become visible when initializing
> another package.

I can only think of some inconsistent inter-package dependencies here.
But this will be covered by the "requires min package version"
dependency management that is already in place in package.el AFAIK.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Achim Gratz <Stromeko@nexgo.de> writes:

> You are missing that the order in which all these things happen becomes
> important.  The newer autoload can only shadow the old one if it's
> processed _before_, likewise initializing other Emacs packages might
> inadvertently pull in old autoloads or definitions before things have
> been rearranged by package.el.  This is not hypothetical, all of that
> has already happened and users have suffered from it.

Yeah, it took me ages to work out. I got lots of very strange and
inconsistent behaviour before I tracked it down to a single (require
'org-html) -- in my own code, incidentally. I'd extended org to do what
I wanted, and didn't know about the exporter update.


> In fact, most of the packages that might be on ELPA already are in their
> own subdirectories and the only thing you'll need to do is not generate
> the first-level autoloads into loaddefs.el and instead have some
> package-autoloads be generated by package.el.  Then create some default
> package configuration with all these packages activated, but give the
> user configuration preference so they can be deactivated.

This is what my branch does, and it's relatively simple to achieve. The
code is messy at the moment, and it also requires some updates to
package.el (because otherwise emacs -q will disable these "core"
packages which it shouldn't). But it is there.


>> I think we both indicated that we want to go this way, we just don't
>> want the Lisp files scattered among dozens of directories, each one
>> with a single file.  Yes, this is a goal that is harder to achieve,
>> but I don't yet see any fundamental difficulties on that path, just
>> some more work.
>
> Single-file packages in core that are duplicated on ELPA (if these even
> exist, I haven't checked) can coexist just fine in a single directory if
> you so desire.  They are orders of magnitude less complicated to deal
> with.  But the total of these should still live in a separate package
> directory that is created just to give them a home.

Not so convinced that this would be good, just because it creates the
edge case of where a single-file package later becomes multi-file
package.

>> I think making such a change incrementally is worse than doing it at
>> once, as far as the directory structure is concerned.  We don't want
>> to change the directory structure several times, ideally not even
>> once.  But if some change is required, it should be done in one go, so
>> we need to decide on the structure up front.
>
> I've said it before and I will say it again just this one time: If Emacs
> takes packetization of the core seriously, then the "hard core" should
> contain just the stuff that is needed for bootstrapping Emacs.
> Everything else should be a package.

That would be my aim too, probably for Emacs-27.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>> (error "org-html is now deprecated")
>> (provide 'org-html)
>>
>>
>> Yes, that would work, but does not provide a good solution to my mind.
>
> Yes, that is one approach.  The other I proposed is using
> after-load-alist:
>
> Extract the example in the attached file.
>
>
>
> - emacs -Q
>
> - Eval the following fragment, after having replaced [path-to] to the
>   proper path where the example was extracted to:
>
>   (push "[path-to]/deprecated-feature-example/foo-package-1.0" load-path)
>   (push "[path-to]/deprecated-feature-example/foo-package-2.0" load-path)
>   (require 'foo-old)
>   => an error will be signaled "foo-old is no longer supported..."
>
> But don't get me wrong.  I really don't think this will be the usual
> case.  In most if not all packages, accounting for such misuses won't be
> required at all.  I just wanted to prove it's not an unsolvable problem.

I like this, and it gives a nice error message of deprecation, and I
think that is good. But this is a solution which requires package
developers to explicitly support declarations of deprecation.

Just removing "food-package-1.0" from `load-path` gives a less nice
error message, but it still fails fast, and requires no developer
support

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Lars Ingebrigtsen]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Of course, with a few heuristics, it would be possible to speed this up.

We could also implement a load-path cache, which would make each lookup
O(1).  Cache invalidation is left as an exercise to the reader.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Lars Ingebrigtsen]

Anyway, I've been trying to follow this discussion, and I just don't
seem to get what the problem is.

Today, a package is (basically) some .el files in a directory.  We want
to extend this paradigm to have in-tree packages that can be updated.

Isn't the obvious solution to have a manifest in each package that says
what .el files are part of that package?

That is, if we have in in-tree package foo, that consists of the files
lisp/foo.el and lisp/image/foo-images.el, then the manifest for foo-0.5
(included in the emacs-26.1 distribution) will be:

'("lisp/foo.el" "lisp/image/foo-images.el")

Now, if the user has updated to a newer version from GNU ELPA, then
those files will land in ~/.emacs.d/elpa/foo, and the manifest will say
that the files belonging to version foo-0.6 will be

'("elpa/foo/foo.el" "elpa/foo/foo-new-images.el" "elpa/foo/foo-me-more.el")

Now, package.el knows that the user wants foo-0.6, so it prepends that
directory to load-path.  In addition, it also has to blacklist all .el
files from previous versions of the package that are visible, which
will, in this case, be the files in the manifest from the build-in
version 0.5.  So it'll add '("lisp/foo.el" "lisp/image/foo-images.el")
to a new variable load-path-blacklist, and saying `(require 'foo-images)'
anywhere will fail with "cannot open load file".

Am I missing something?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>
>> But don't get me wrong.  I really don't think this will be the usual
>> case.  In most if not all packages, accounting for such misuses won't be
>> required at all.  I just wanted to prove it's not an unsolvable problem.
>
> I like this, and it gives a nice error message of deprecation, and I
> think that is good. But this is a solution which requires package
> developers to explicitly support declarations of deprecation.

Indeed.  But in case a package offers a public feature/interface to be
used by anybody, including non-package code, such an explicit handling
would be TRTTD, IMO.

> Just removing "food-package-1.0" from `load-path` gives a less nice
> error message, but it still fails fast, and requires no developer
> support

True.  But I don't think we have to account for all possible "misuses"
and bugs.  We all run into issues.  But we have to try to not get biased
by a specific problem one once had and which might have taken hours to
fix, if after all it's clearly a "usage error" or -- depending on how
you see it -- in your case a deprecation of a public functionality in a
package that wasn't flagged as deprecated in the first place, before
actually having been removed.  Emacs is more like a white box.  You will
always find dozen of ways to break something.  But we shall not
primarily focus on such edge-case scenarios, I think.

And I think this discussion has nothing to do with the directory layout.
It's a different topic IMO, though you try to fix it with the directory
layout.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Anyway, I've been trying to follow this discussion, and I just don't
> seem to get what the problem is.

Me too.

> Now, package.el knows that the user wants foo-0.6, so it prepends that
> directory to load-path.  In addition, it also has to blacklist all .el
> files from previous versions of the package that are visible, which
> will, in this case, be the files in the manifest from the build-in
> version 0.5.  So it'll add '("lisp/foo.el" "lisp/image/foo-images.el")
> to a new variable load-path-blacklist, and saying `(require 'foo-images)'
> anywhere will fail with "cannot open load file".

Do you really see a need for doing this?  Either is foo-images some
package-internal feature.  Then nobody should care about it and no one
should ever require it beside the old version of the package itself.  Or
it belongs to a public "API" that clients (whether packages or not) may
use at will.  But then foo package should account for it and somehow
explicitly mark it as deprecated before removing the feature/file in a
v1+n version.  (Maybe package.el should support explicit withdrawal of a
formerly published public feature.  I'm not sure if a solution at the
load-path "low-"level would be TRTTD.)

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Lars Ingebrigtsen]

Alain Schneble <a.s@realize.ch> writes:

> Do you really see a need for doing this?  Either is foo-images some
> package-internal feature.  Then nobody should care about it and no one
> should ever require it beside the old version of the package itself.

"Should" is one of those words that should be avoided.  :-)

People load bits and bobs of things in their .emacs files, and it is a
problem in practice, according to the Org people.

And it seems like a trivial problem to fix.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: feature/integrated-elpa 4f6df43 15/23: README added

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

>>>>> "AG" == Achim Gratz <Stromeko@nexgo.de> writes:

AG> If Emacs takes packetization of the core seriously, then the "hard core"
AG> should contain just the stuff that is needed for bootstrapping Emacs.
AG> Everything else should be a package. Single-file packages would live in
AG> one directory together, and multi-file packages would be in their own
AG> directory each.

We may get there, but not as our first step. Let me reiterate our plan:

The purpose of the first step is *only* to make ELPA first-class, so that a
package can live primarily in ELPA, but *appear to the end user* as if it had
been developed alongside Emacs. This removes any privileged distinction that
packages living in Emacs.git may have, and encourages future contributions to
move to ELPA. Other than this, there should be no substantive changes.

This will likely require changes to package.el, so that a package distributed
in lisp/ can be updated using package.el. These are the only changes to
package.el that should be necessary initially.

What Emacs developers will notice after this step is that several packages
leave Emacs.git and move to ELPA.git, or back to their points of origin with
release versions in ELPA.git. Nothing at all should change for Emacs packages
that don't move to ELPA.

All this talk about packagizing Emacs.git, so we can manage everything with
package.el (including building and autoloads-generation), and keep everything
into its own directory, is putting the cart before the horse. We're not ready
for that step yet. First, let's make ELPA more useful, then we can talk about
reorganizing core Emacs once this initial phase has seen successful reception
From users.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Alain Schneble <a.s@realize.ch> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>> I like this, and it gives a nice error message of deprecation, and I
>> think that is good. But this is a solution which requires package
>> developers to explicitly support declarations of deprecation.
>
> Indeed.  But in case a package offers a public feature/interface to be
> used by anybody, including non-package code, such an explicit handling
> would be TRTTD, IMO.

Yes, I think that is true. I don't think that there is explicit support
for this form of deprecation in Emacs at the moment. We should write a
package and add it to ELPA!



>> Just removing "food-package-1.0" from `load-path` gives a less nice
>> error message, but it still fails fast, and requires no developer
>> support
>
> True.  But I don't think we have to account for all possible "misuses"
> and bugs.  We all run into issues.  But we have to try to not get biased
> by a specific problem one once had and which might have taken hours to
> fix

I try not to. It's just a problem that packages in core having when
upgrading from ELPA. It's a clear, specific. I've had it, true, and it
took me a lot of time to work out, but as Achim says so have other people.


> And I think this discussion has nothing to do with the directory layout.
> It's a different topic IMO, though you try to fix it with the directory
> layout.

It is a problem that a package.el based solution would not have, and
which it achieves through in a simple and straight-forward way, though
it's use of it's directory layout.

Personally, I consider the directory layout to be a side issue -- direct
use of package.el is the main issue. It's there, Emacs already supports
in for packages in ~/.emacs.d, and in site lisp, it's already plumbed
into startup and all the files in ELPA are already built with package.el
in mind.

To me using it is a slam dunk, as our friends over the pond like to say.
I am not at all surprised that people objected to my idea of introducing
a new top level directory is contentious -- that is why I asked the
question in the first place; I am a little surprised at the objections
to a per-package directory structure for these packages. So far, I have
only understood two objections: Drew's and Stefan's. Why would it be a
problem?

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Of course, with a few heuristics, it would be possible to speed this up.
>
> We could also implement a load-path cache, which would make each lookup
> O(1).  Cache invalidation is left as an exercise to the reader.

There are two hard problems in computer science. Cache invalidation and
coming up with good names. My solution is O(1) and doesn't even require
the initial scan of all files, though.

It seems not to be relevant, though, as we're not going to have multiple
directories.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Anyway, I've been trying to follow this discussion, and I just don't
> seem to get what the problem is.
>
> Today, a package is (basically) some .el files in a directory.  We want
> to extend this paradigm to have in-tree packages that can be updated.

I don't. I want to have a parallel tree of in package.el format.
Updating (or, rather, outdating since we are not changing files) comes
for free.


> Isn't the obvious solution to have a manifest in each package that says
> what .el files are part of that package?
>
> That is, if we have in in-tree package foo, that consists of the files
> lisp/foo.el and lisp/image/foo-images.el, then the manifest for foo-0.5
> (included in the emacs-26.1 distribution) will be:
>
> '("lisp/foo.el" "lisp/image/foo-images.el")
>
> Now, if the user has updated to a newer version from GNU ELPA, then
> those files will land in ~/.emacs.d/elpa/foo, and the manifest will say
> that the files belonging to version foo-0.6 will be
>
> '("elpa/foo/foo.el" "elpa/foo/foo-new-images.el" "elpa/foo/foo-me-more.el")

Yes, although, this is not just the .el files. We also have the info and
etc files which need similar treatment.


> Now, package.el knows that the user wants foo-0.6, so it prepends that
> directory to load-path.  In addition, it also has to blacklist all .el
> files from previous versions of the package that are visible, which
> will, in this case, be the files in the manifest from the build-in
> version 0.5.  So it'll add '("lisp/foo.el" "lisp/image/foo-images.el")
> to a new variable load-path-blacklist, and saying `(require 'foo-images)'
> anywhere will fail with "cannot open load file".

Also, loaddefs.el. Currently, the in-tree foo-0.5 will have it's
autoloads dumbed into loaddefs.el (unless it's one of those packages
with it's own loaddefs). We'd also need to undefine all the autoloads
defined in there which relate to foo-0.5.

I think that there might be an issue with timing also. Say some one has
a .emacs like so:

(require 'foo)
(package-initialize)

Then I think that they will get foo-0.5 because package.el hasn't done
all the things you mention yet. Thinking quickly, this is an issue for
my solution also, and will require a careful solution.

> Am I missing something?

You have thought of most of the issues that came to mind for an in-tree
solution; we probably are both missing things. We won't know till it's
build.

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

> All this talk about packagizing Emacs.git, so we can manage everything with
> package.el (including building and autoloads-generation), and keep everything
> into its own directory, is putting the cart before the horse. We're not ready
> for that step yet. First, let's make ELPA more useful, then we can talk about
> reorganizing core Emacs once this initial phase has seen successful reception
> From users.

To be clear, I never suggested re-organizing the current files in Emacs
-- in fact, I suggested leaving it *exactly* as it is. I just suggested
supplementing it one part of the build, and some changes to package.el
initialisation.

The in tree solution, conversely, requires copying files into lisp/,
adding more non-source files into this tree in an in-directory source
build of emacs. And, as Lars described nicely, also requires generation
of manifests, and changes to the load-path mechanism in C.

Okay. That's really it. Please, John, if I ever post on this topic
again, can you kick me off the mailing list?

Phil

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Wed, 19 Oct 2016 20:26:33 +0200
> 
> Eli Zaretskii writes:
> > I don't see why restarting won't be a solution.  If load-path was
> > re-arranged to put the latest version of a package first, and the
> > package's autoloads are on a file that has been regenerated, what else
> > is missing to make restart the correct solution?
> 
> You are missing that the order in which all these things happen becomes
> important.  The newer autoload can only shadow the old one if it's
> processed _before_, likewise initializing other Emacs packages might
> inadvertently pull in old autoloads or definitions before things have
> been rearranged by package.el.

I understand that.  I just don't see why we couldn't do this in the
correct order, given that the meta-data which accompanies the update
of a package is correct.

> > If load-path is rearranged when a newer version of Org is installed,
> > and org-loaddefs are regenerated, then, after a restart, any 'load'
> > and 'require' will find the new files first, and the old ones will be
> > effectively invisible to Emacs.  Right?
> 
> If you only want to look at that one-dimensional example, yes.  Real
> Emacs installations are quite a bit more varied.  The tools we have
> today to sove this problem make things more brittle when they should
> become more robust.

I understand that in general having some bundled packages in another
repository makes things more complex, perhaps much more complex.  But
since the majority wants that, we will have to deal with that
complexity and solve it well enough to work in practice.

> I still don't get what you hope to gain from throwing things that don't
> belong together (and aren't upstrem) in a single directory (only talking
> about the code parts here, documentation can be pulled someplace else).

Simplicity of use for everyone and transparency for end users (who
shouldn't be concerned in which Git repo a given package is
maintained).

> In fact, most of the packages that might be on ELPA already are in their
> own subdirectories and the only thing you'll need to do is not generate
> the first-level autoloads into loaddefs.el and instead have some
> package-autoloads be generated by package.el.  Then create some default
> package configuration with all these packages activated, but give the
> user configuration preference so they can be deactivated.

I see no reasons that this couldn't be done without having a separate
directory per package.

> Single-file packages in core that are duplicated on ELPA (if these even
> exist, I haven't checked) can coexist just fine in a single directory if
> you so desire.  They are orders of magnitude less complicated to deal
> with.  But the total of these should still live in a separate package
> directory that is created just to give them a home.

I see no reasons to have a separate directory for packages that come
bundled.

> I've said it before and I will say it again just this one time: If Emacs
> takes packetization of the core seriously, then the "hard core" should
> contain just the stuff that is needed for bootstrapping Emacs.
> Everything else should be a package.

If you mean that those "packages" are downloaded by end users when
they install Emacs, and don't come with the release tarball, then this
is against the current agreements, AFAIU.  The current agreements are
that the ELPA packages are downloaded and installed in the Emacs tree
when the release is being tarred, and come in the bundle ready to be
used.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Wed, 19 Oct 2016 21:24:54 +0200
> 
> Alain Schneble writes:
> > Of course.  The order of directories in load path is always relevant.
> > Whatever structure we choose.
> 
> Good.  Now take that last step: if you want to be sure that you don't
> load something you don't want to be loaded, it must not be found via
> load-path at all times during the lifetime of the Emacs process.

That is inaccurate.  The accurate description would be "it must either
not be found via load-path, or not mentioned in any 'load' and
'require' form and in any autoload cookie.

> That precludes almost all solutions that depend on removal or
> shadowing during later steps in the initialization sequence and
> where not-yet-to-be-initialized packages become visible when
> initializing another package.

I don't see how you jump to that conclusion, except by assuming some
buggy or chaotic behavior on some other packages.

But we are not talking here about some obscure package downloaded from
some github spot.  We are talking about packages that today live in
core, and will (or at least should) still get the same amount of
attention from the Emacs developers when they are maintained on ELPA
as they do today.  So any assumptions of gross bugs like the ones you
evidently have in mind in those packages are simply invalid.  I
understand you have bumped into such problems, and I understand that
they could happen, but I disagree that we should account for them
happening in the packages that are the subject of this discussion for
longer than a few hours, exactly like we have today with any commit
that breaks some build somewhere.

Errors do happen, but if they are fixed soon enough, the design
doesn't have to take these errors into consideration, and doesn't have
to be robust against them.  (It is, of course, nice to have such
robustness where it comes for free or for a very low price, but not
where the price is high.)

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: Alain Schneble <a.s@realize.ch>
> Date: Wed, 19 Oct 2016 22:13:09 +0200
> Cc: emacs-devel@gnu.org
> 
> >                                                                   That
> > precludes almost all solutions that depend on removal or shadowing
> > during later steps in the initialization sequence and where
> > not-yet-to-be-initialized packages become visible when initializing
> > another package.
> 
> I can only think of some inconsistent inter-package dependencies here.
> But this will be covered by the "requires min package version"
> dependency management that is already in place in package.el AFAIK.

Yes, updating a package should also update all of its dependencies.
And it should bail out if some of the dependencies are not yet
available (e.g., if some dependency package was not yet update to
account for the changes in the main package).

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Wed, 19 Oct 2016 20:47:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> If everybody does the right thing. If not everybody does the right
> >> thing, the old version gets loaded. Happened to me, it's hard to debug
> >> and hard to work out what is going wrong.
> >
> > OK, so we agree that if everybody does the right thing, this will
> > work.  That's all we need at this point, because we certainly don't
> > need to cater to packages which do the wrong thing: such packages will
> > be fixed in no time, because they are bundled and are part of any
> > Emacs installation.
> 
> We should cater to packages which do the wrong thing, by ensuring that
> the break quickly and obviously.

No, we most definitely don't!  We are talking about packages that are
conceptually part of the Emacs core, even if they are maintained in
another repository.  Those packages will get all the attention they
should from the Emacs developers, so we can safely assume this kind of
problems will never happen in them, except by mistake that is
corrected soon enough after it happens, like any other inadvertent
breakage in Emacs.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>,  <Stromeko@nexgo.de>,  <emacs-devel@gnu.org>
> Date: Wed, 19 Oct 2016 21:24:03 +0100
> 
> > - emacs -Q
> >
> > - Eval the following fragment, after having replaced [path-to] to the
> >   proper path where the example was extracted to:
> >
> >   (push "[path-to]/deprecated-feature-example/foo-package-1.0" load-path)
> >   (push "[path-to]/deprecated-feature-example/foo-package-2.0" load-path)
> >   (require 'foo-old)
> >   => an error will be signaled "foo-old is no longer supported..."
> >
> > But don't get me wrong.  I really don't think this will be the usual
> > case.  In most if not all packages, accounting for such misuses won't be
> > required at all.  I just wanted to prove it's not an unsolvable problem.
> 
> I like this, and it gives a nice error message of deprecation, and I
> think that is good. But this is a solution which requires package
> developers to explicitly support declarations of deprecation.

If this is what we want, I see no reason to require such support in
the packages that will be bundled.  They are Emacs core packages for
all purposes, so we are requiring this from ourselves.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Wed, 19 Oct 2016 20:57:22 +0100
> Cc: emacs-devel@gnu.org
> 
> > If we take the files bundled with Emacs and look at how many ELPA
> > packages they would turn into, we'd end up with a load path that has
> > more than 200 elements.
> 
> My load-path has 151 at the moment.

Mine has just 25.  I hope this explains to you why I'm opposed to
making load-path much longer.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Michael Albinus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> The in tree solution, conversely, requires copying files into lisp/,
> adding more non-source files into this tree in an in-directory source
> build of emacs. And, as Lars described nicely, also requires generation
> of manifests, and changes to the load-path mechanism in C.

Packages carry also other files, like *.texi. Your proposal does not
handle this. And copying them to doc/misc, as one would expect, breaks
your philosophy to have them all-in-one-subdirectory-below-lisp/.

The current elpa approach does not handle this problem either, but
that's another problem.

> Phil

Best regards, Michael.

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Alain Schneble <a.s@realize.ch> writes:
>
>> Indeed.  But in case a package offers a public feature/interface to be
>> used by anybody, including non-package code, such an explicit handling
>> would be TRTTD, IMO.
>
> Yes, I think that is true. I don't think that there is explicit support
> for this form of deprecation in Emacs at the moment. We should write a
> package and add it to ELPA!

Good idea. Why not.  I just doubt that many packages will ever require
this.

>>> Just removing "food-package-1.0" from `load-path` gives a less nice
>>> error message, but it still fails fast, and requires no developer
>>> support
>>
>> True.  But I don't think we have to account for all possible "misuses"
>> and bugs.  We all run into issues.  But we have to try to not get biased
>> by a specific problem one once had and which might have taken hours to
>> fix
>
> I try not to. It's just a problem that packages in core having when
> upgrading from ELPA. It's a clear, specific. I've had it, true, and it
> took me a lot of time to work out, but as Achim says so have other people.

Ok.

>> And I think this discussion has nothing to do with the directory layout.
>> It's a different topic IMO, though you try to fix it with the directory
>> layout.
>
> It is a problem that a package.el based solution would not have, and
> which it achieves through in a simple and straight-forward way, though
> it's use of it's directory layout.

Alright.  But look, no one prevents us from removing the old version
from load-path, for packages that have their own directory.  Large
packages happen to have their own directory anyway.  For single file
packages, the issue is non-existent, as renaming such a file is not
really a use case as it would essentially introduce a new package
(name).  And then there may be a bunch of (smaller) multi-file packages
that could run into such a "deleted-file-issue".  But keep in mind that
all bundled core/tarball packages are from a trusted source and we can
circumvent or fix these problems (in the worst case with explicit
mechanism we have discussed).  It's in Emacs control.

So I think that legitimating a separate directory for each structure
using this edge case sounds like nit-picking.

> Personally, I consider the directory layout to be a side issue -- direct
> use of package.el is the main issue. It's there, Emacs already supports
> in for packages in ~/.emacs.d, and in site lisp, it's already plumbed
> into startup and all the files in ELPA are already built with package.el
> in mind.

It's there and we fortunately can extend it.

> To me using it is a slam dunk, as our friends over the pond like to say.
> I am not at all surprised that people objected to my idea of introducing
> a new top level directory is contentious -- that is why I asked the
> question in the first place; I am a little surprised at the objections
> to a per-package directory structure for these packages. So far, I have
> only understood two objections: Drew's and Stefan's. Why would it be a
> problem?

IMHO, trying to implement the simplest possible file structure and not
introducing a new one but reusing the existing one, are reasons enough.
Also, having a dual directory layout in released Emacs tarballs doesn't
feel good to me.  And to my understanding, we haven't seen a road block
with the "integrated" ./lisp layout so far.

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Alain Schneble]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Alain Schneble <a.s@realize.ch> writes:
>
>> Do you really see a need for doing this?  Either is foo-images some
>> package-internal feature.  Then nobody should care about it and no one
>> should ever require it beside the old version of the package itself.
>
> "Should" is one of those words that should be avoided.  :-)

You are so right.  I should learn to substitute it with better ones; I
know I should improve my English skills. :)

> People load bits and bobs of things in their .emacs files, and it is a
> problem in practice, according to the Org people.
>
> And it seems like a trivial problem to fix.

I just don't think it's worth to fix it on that level upfront.  See also
my thoughts here, second half:
https://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00603.html

Alain

Re: feature/integrated-elpa 4f6df43 15/23: README added

[Achim Gratz]

Lars Ingebrigtsen writes:
[…]
> Am I missing something?

You have only solved the obvious load-path dependent problems yet (todo:
autoloads and customizations, dealing with already loaded, but outdated
packages and dependecies), plus you've introduced a huge technical
liability: you can no longer be sure whether or not a file that's in
load-path will actually be loaded when you try to.  This now at least
depends on whether it is mentioned in one of many manifest files,
whether they've been correctly processed, etc.pp.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves