user-controlled load-path extension: load-dir

user-controlled load-path extension: load-dir

[Ted Zlatanov]

Sorry if this has been discussed before.  I don't recall such a
discussion and the search terms are too generic.

I think it's useful to load on startup any .el files placed in a
particular directory (if the user approves the directory).  For example,
if the user sets `load-dir' to '("~/.emacs.d/load"),
~/.emacs.d/load/*.el will be loaded on startup.  The `load-path' is not
involved, this would be a straight `eval' call.

`load-dir' is just a name suggestion.

This can be done on the user side, of course, but if Emacs had a
built-in facility to do it, enabling and controlling Emacs
configurations in a modular way would be easier, especially for new
users.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ted Zlatanov <tzz@lifelogs.com> writes:
> I think it's useful to load on startup any .el files placed in a
> particular directory (if the user approves the directory).  For example,
> if the user sets `load-dir' to '("~/.emacs.d/load"),
> ~/.emacs.d/load/*.el will be loaded on startup.  The `load-path' is not
> involved, this would be a straight `eval' call.

The same way we have user-init-file, I too would welcome Emacs to offer
a default user-load-path that's known to be e.g. ~/.emacs.d/load.d.
That would allow third party elisp code to just drop a file in there and
have users all set to use a new facility.

For example, I would use that for el-get and bootstrapping it into the
user's setup would become easier, and would allo to avoid having to edit
user-init-file.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Thu, Mar 03 2011, Dimitri Fontaine wrote:

> The same way we have user-init-file, I too would welcome Emacs to offer
> a default user-load-path that's known to be e.g. ~/.emacs.d/load.d.
> That would allow third party elisp code to just drop a file in there and
> have users all set to use a new facility.

Especially if Emacs was able to build and load an autoload file for the
files placed in that directory. The user would have *nothing* to do to
use a new feature. Not even touching its `user-init-file'.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Julien> Especially if Emacs was able to build and load an autoload file for the
Julien> files placed in that directory. The user would have *nothing* to do to
Julien> use a new feature. Not even touching its `user-init-file'.

package.el does this, more or less.  It doesn't just load all files, but
does some version and dependency checks.  But, e.g., it is trivial for
el-get to be supplied as a package.  This feature is enabled by default
in Emacs 24.

Tom

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 10:18:03 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Julien> Especially if Emacs was able to build and load an autoload file for the
Julien> files placed in that directory. The user would have *nothing* to do to
Julien> use a new feature. Not even touching its `user-init-file'.

Tom> package.el does this, more or less.  It doesn't just load all files, but
Tom> does some version and dependency checks.  But, e.g., it is trivial for
Tom> el-get to be supplied as a package.  This feature is enabled by default
Tom> in Emacs 24.

OK, but there's no general facility to do this.  If I want to bundle up
some code and load it on startup, I have to modify the load-path and my
startup .emacs right now (which encourages monolithic configurations and
other bad practices).  I don't think package.el should be required to do
this kind of modularization.

As far as el-get, I guess it could be made into a package, but it's a
package manager and has to work in cases where package.el is not
available or the user doesn't want to use it.  I don't think el-get is
the only use for the general load-dir facility, in any case.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> package.el does this, more or less.  It doesn't just load all files, but
> does some version and dependency checks.  But, e.g., it is trivial for
> el-get to be supplied as a package.  This feature is enabled by default
> in Emacs 24.

Would it be accepted, though?  If yes, I'd better go begin those FSF
paperwork.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Tom> package.el does this, more or less.  It doesn't just load all files, but
Tom> does some version and dependency checks.  But, e.g., it is trivial for
Tom> el-get to be supplied as a package.  This feature is enabled by default
Tom> in Emacs 24.

Dimitri> Would it be accepted, though?  If yes, I'd better go begin those FSF
Dimitri> paperwork.

Suppose Emacs did add a "load all the files in a directory" feature.
You would tell your users "download this and install it here".
Or you would supply some elisp on a web page for them to eval, and this
elisp would do the downloading.

Well, you can do the exact same things with a package.  You don't need
it to be in elpa.gnu.org.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> Suppose Emacs did add a "load all the files in a directory" feature.
> You would tell your users "download this and install it here".
> Or you would supply some elisp on a web page for them to eval, and this
> elisp would do the downloading.
>
> Well, you can do the exact same things with a package.  You don't need
> it to be in elpa.gnu.org.

What el-get currently does not do is editing any user's file.  What's
proposed in this thread is a way for el-get to bootstrap itself in a
user's setup without having to edit the user's init file.

Now, this would be answered by a known user's default load-path.  Julien
and Ted where adding to it actual loading then autoloads, I would live
without those.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 20:45:45 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

DF> What's proposed in this thread is a way for el-get to bootstrap
DF> itself in a user's setup without having to edit the user's init
DF> file.

s/el-get/any Emacs Lisp code/

The only reason I can think why this would not be acceptable is
security.  Otherwise this makes a lot of sense.

For security I would propose checksumming any code in the load-dir and
when new files are added or existing files are changed, query the user
and store a file checksum in an alist.

I'm willing to make this facility a package that users can install.  But
I think it's better to put it in the Emacs core so it's available by
default and without special action from the user beyond enabling a
single variable.  I feel strongly it would enhance Emacs to provide this.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ted Zlatanov <tzz@lifelogs.com> writes:
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.

My answer there would be to just add to load-path, but have no automatic
mechanism to load the files in there or register the autoloads.  Having
a known place where to download el-get.el and being able to tell user to
just (require 'el-get) is all I want here.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]


On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
> 
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.
> 
> For security I would propose checksumming any code in the load-dir and
> when new files are added or existing files are changed, query the user
> and store a file checksum in an alist.

This would have to happen the first time the user loaded any package, or you have no security. If you do this the first time the user has loaded any package, you have removed the `automatically' part of your proposed system, which is the only real advantage it has.

The only way your proposal seems (to me) to be an improvement is if we train users to ignore the security questions and just always click on `I agree'.  I hope you agree that that isn't a good idea.

Modern users have been trained to expect `Extensions' and `Add-Ons', and don't seem to mind picking from a list (for things like FireFox, a very large list with many overlapping choices). I think that's a better user model to follow.

I hope that helps,
*Chad


[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 4 Mar 2011 12:25:22 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
>> 
>> The only reason I can think why this would not be acceptable is
>> security.  Otherwise this makes a lot of sense.
>> 
>> For security I would propose checksumming any code in the load-dir and
>> when new files are added or existing files are changed, query the user
>> and store a file checksum in an alist.

CB> This would have to happen the first time the user loaded any package,
CB> or you have no security. If you do this the first time the user has
CB> loaded any package, you have removed the `automatically' part of your
CB> proposed system, which is the only real advantage it has.

No.  This query is only for Lisp code placed in the load-dir, not for
packages in general.  So you'd have the query as part of the el-get
installation for instance and it wouldn't show up again.  Compare with
Mac OS X which does this for new applications you've downloaded from the
net, alerting the very first time.

CB> The only way your proposal seems (to me) to be an improvement is if we
CB> train users to ignore the security questions and just always click on
CB> I agree'.  I hope you agree that that isn't a good idea.

It's not a good idea, but it's not what I'm describing either.  This
query should show up very rarely, not all the time.  It's meant for
bootstrapping and for user code.  If more than one file needs to be
approved the user could use `Y' instead of `y', but in any case the
query would be very rare.

CB> Modern users have been trained to expect `Extensions' and `Add-Ons',
CB> and don't seem to mind picking from a list (for things like FireFox, a
CB> very large list with many overlapping choices). I think that's a
CB> better user model to follow.

Let's take an example.  I have 10 pieces of code related to Emacs I want
to load.  Do I have to explicitly list them in my .emacs?  Or can I just
put them in a directory?  The former is twice the work since I have to
put them in some directory anyhow and puts the synchronization burden on
me.  So I can write some Lisp to do the discovery and load them
automatically... which is what I'm offering for everyone's benefit (with
some security features).  I already do this for myself so I can propose
a patch or a package quickly :)

Package management is not what I'm proposing.  The package manager
*installation* would write a bootstrapper piece of code that has to be
approved *once*.  So while the load-dir is useful for installing el-get,
I don't think el-get is the only reason to adopt it.

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 4, 2011, at 12:46 PM, Ted Zlatanov wrote:

> No.  This query is only for Lisp code placed in the load-dir, not for
> packages in general.  So you'd have the query as part of the el-get
> installation for instance and it wouldn't show up again.  Compare with
> Mac OS X which does this for new applications you've downloaded from the
> net, alerting the very first time.

If the user has to go through a set of steps to install and activate a package, how isn't that what ELPA does?

As I'm understanding you, any magic directory is either no less work for the user or has unforgivable security issues.  What am I missing?

*Chad

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]


On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
> 
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.
> 
> For security I would propose checksumming any code in the load-dir and
> when new files are added or existing files are changed, query the user
> and store a file checksum in an alist.

This would have to happen the first time the user loaded any package, or you have no security. If you do this the first time the user has loaded any package, you have removed the `automatically' part of your proposed system, which is the only real advantage it has.

The only way your proposal seems (to me) to be an improvement is if we train users to ignore the security questions and just always click on `I agree'.  I hope you agree that that isn't a good idea.

Modern users have been trained to expect `Extensions' and `Add-Ons', and don't seem to mind picking from a list (for things like FireFox, a very large list with many overlapping choices). I think that's a better user model to follow.

I hope that helps,
*Chad


[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> What el-get currently does not do is editing any user's file.  What's
Dimitri> proposed in this thread is a way for el-get to bootstrap itself in a
Dimitri> user's setup without having to edit the user's init file.

Yes, my proposal accomplishes that.  There are two problems here.

The first problem is, how does the user ever get el-get.el?
Do they download it from a web site?  Do you have some installer elisp
on a web site to eval (this is what I did with ELPA)?  Etc.

Whatever the answer is to that problem, there is a similar
package.el-based answer.

E.g., if you have some elisp on your web site to download and install
el-get.el, just write different elisp (untested):

(let ((buffer (url-retrieve-synchronously "the url/el-get.el")))
  (with-current-buffer buffer
    ;; strip the headers
    (re-search-forward "^$" nil 'move)
    (forward-char)
    (delete-region (point-min) (point))
    (package-install-from-buffer (package-buffer-info) 'single))
  (kill-buffer buffer))

If users just download your file by hand, then they just do that and
then M-x package-install-file.


The second problem is, how does el-get.el get activated?

If el-get.el is an ordinary package, in the package.el sense, then in
Emacs 24 it will just work with zero changes to the user's init file.
Autoloads will be extracted, etc.  You can make it do any
el-get-specific activation quite easily using an autoload comment.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> Whatever the answer is to that problem, there is a similar
> package.el-based answer.
>
> E.g., if you have some elisp on your web site to download and install
> el-get.el, just write different elisp (untested):

Here's the code to copy paste and run to install el-get currently.

(url-retrieve
 "https://github.com/dimitri/el-get/raw/master/el-get-install.el"
 (lambda (s)
   (end-of-buffer)
   (eval-print-last-sexp)))

>     (package-install-from-buffer (package-buffer-info) 'single))

I can see your point about el-get being a package even if not included
in ELPA by default, now that package.el is part of Emacs24.  That leaves
any user of Emacs23 with no solution though.  I will consider that at
some point I guess, but currently el-get allows you to get package.el in
emacs23 :)

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> I can see your point about el-get being a package even if not
Dimitri> included in ELPA by default, now that package.el is part of
Dimitri> Emacs24.  That leaves any user of Emacs23 with no solution
Dimitri> though.  I will consider that at some point I guess, but
Dimitri> currently el-get allows you to get package.el in emacs23 :)

If somebody added the feature you want to Emacs, presumably it would
only be in Emacs 24, leaving you in the same situation.

Or maybe the proposal is to put it into Emacs 23.  But then we might as
well just put package.el into Emacs 23.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> If somebody added the feature you want to Emacs, presumably it would
> only be in Emacs 24, leaving you in the same situation.

Point.  I would still prefer to depend on user-load-path or something,
it strikes me as more general: you don't have to edit existing scripts.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> Point.  I would still prefer to depend on user-load-path or something,
Dimitri> it strikes me as more general: you don't have to edit existing scripts.

Now you're just looking for reasons to avoid it :) 

Tom

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 22:33:35 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

DF> Tom Tromey <tromey@redhat.com> writes:
>> If somebody added the feature you want to Emacs, presumably it would
>> only be in Emacs 24, leaving you in the same situation.

DF> Point.  I would still prefer to depend on user-load-path or something,
DF> it strikes me as more general: you don't have to edit existing scripts.

Tom is talking about one thing (package management) and we're talking
about another (generic Lisp snippet management).  Sure, you can do the
latter with the former, but it's a long stretch and is fighting the
user.  Look, if I just want to put a file with

(setq myvar xyz)

or the el-get initialization, or whatever in the user-load-path, why do
I have to make a package out of it?

By analogy consider some of the software that lets you put a snippet in
a conf.d directory, obviously implying that this is convenient for the
user.  This is just a sampling from my machine:

apache2
newer crons (/etc/cron.d)
AppArmor
Grub
libpaper
sudo
logrotate
rsyslog
modprobe
sane
PAM

Why not provide the same level of convenience in Emacs?  Other than
security, what's the argument against it?  We all understand the
benefits of modularization made easy, right?

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]


On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
> 
> By analogy consider some of the software that lets you put a snippet in
> a conf.d directory, obviously implying that this is convenient for the
> user.  This is just a sampling from my machine:

Those are all machine-wide configurations.  For that, use site-lisp.

*Chad

[-- Attachment #2: Type: text/html, Size: 639 bytes --]

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

On Fri, Mar 04, 2011 at 09:18:22PM -0600, Ted Zlatanov wrote:
> On Fri, 04 Mar 2011 22:33:35 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 
> 
> DF> Tom Tromey <tromey@redhat.com> writes:
> >> If somebody added the feature you want to Emacs, presumably it would
> >> only be in Emacs 24, leaving you in the same situation.
> 
> DF> Point.  I would still prefer to depend on user-load-path or something,
> DF> it strikes me as more general: you don't have to edit existing scripts.
> 
> Tom is talking about one thing (package management) and we're talking
> about another (generic Lisp snippet management).  Sure, you can do the
> latter with the former, but it's a long stretch and is fighting the
> user.  Look, if I just want to put a file with
> 
> (setq myvar xyz)
> 
> or the el-get initialization, or whatever in the user-load-path, why do
> I have to make a package out of it?
> 
> By analogy consider some of the software that lets you put a snippet in
> a conf.d directory, obviously implying that this is convenient for the
> user.  This is just a sampling from my machine:
> 
> apache2
> newer crons (/etc/cron.d)
> AppArmor
> Grub
> libpaper
> sudo
> logrotate
> rsyslog
> modprobe
> sane
> PAM
> 
> Why not provide the same level of convenience in Emacs?  Other than
> security, what's the argument against it?  We all understand the
> benefits of modularization made easy, right?
> 
> Ted
> 
> 

There are ways to solve the problem you are looking at without wiring more
logic into Emacs. Use the .emacs file as a more sophisticated loader for a
complex configuration when necessary.

I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
I would look at these solutions first before proposing code that has to
be wired into the bootstrap codepath.

There are a number of issues a reliable loader should address:

how does it handle errors ?

how does it handle --batch ?

how does it handle --daemon ?

how simple,transparent,debuggable is the loader code ?

I have not seen any code so far in your proposal unless I accidentally
skipped or deleted a message with it.






[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Sat, 5 Mar 2011 11:11:34 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
>> 
>> By analogy consider some of the software that lets you put a snippet in
>> a conf.d directory, obviously implying that this is convenient for the
>> user.  This is just a sampling from my machine:

CB> Those are all machine-wide configurations.  For that, use site-lisp.

Bazaar has a plugins directory; files in it are automatically activated,
as an example of a user-level facility like this.  Anyhow, my point is
that placing a file in a directory is inherently more modular and
convenient to the user than augmenting a single file.  Do you disagree?

CB> If the user has to go through a set of steps to install and activate a
CB> package, how isn't that what ELPA does?

They are not packages, they are snippets of code.  ELPA requires far
more structure and many more steps.  For what I'm proposing, a lot less
work is required (just a y/n prompt the first time a snippet is found to
ensure it's not placed there maliciously).

On Sat, 5 Mar 2011 23:21:51 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> There are ways to solve the problem you are looking at without wiring more
MM> logic into Emacs. Use the .emacs file as a more sophisticated loader for a
MM> complex configuration when necessary.

I can and do, sure.  I don't think it creates a nice user experience.

MM> I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
MM> I would look at these solutions first before proposing code that has to
MM> be wired into the bootstrap codepath.

Thanks for pointing those out.  Would Grail or tidyconfig work as a core
option on startup (meaning the user doesn't have to install a package,
just customize a boolean)?  Do they handle signing the snippets the
first time they are found?  Can you give us a comparison of them and any
other similar solutions you know?  They have to be GPL and assigned to
Emacs to be included.

MM> There are a number of issues a reliable loader should address:

MM> how does it handle errors ?

MM> how does it handle --batch ?

MM> how does it handle --daemon ?

MM> how simple,transparent,debuggable is the loader code ?

MM> I have not seen any code so far in your proposal unless I accidentally
MM> skipped or deleted a message with it.

I can write the code but haven't seen the need for actual code yet (I
could have just comitted some code to Emacs instead, but find that
option to be antisocial).

I'd rather use something that already exists such as Grail or tidyconfig
that you kindly pointed out.  If you could tell us how any potential
solutions can help answer the questions above, that would be wonderful.
I can look at any solutions and evaluate them if you like.

To answer your questions from my perspective, IMHO a loader should work
like package.el.  IOW, whenever and however package.el is loaded
currently, this loader we're discussing should also be loaded, since
it's effectively a package manager but with snippets instead of
packages.

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 7, 2011, at 8:24 AM, Ted Zlatanov wrote:

> Bazaar has a plugins directory; files in it are automatically activated,
> as an example of a user-level facility like this.  Anyhow, my point is
> that placing a file in a directory is inherently more modular and
> convenient to the user than augmenting a single file.  Do you disagree?

My point is that the user must take an explicit step to do either, and
that placing a file in a magic directory and then answering questions
about it (both steps are required) is significantly less convenient
and equally or less modular than running an installer command.

So, yes, I do disagree.  What you call `augmenting a single file' can
be done with a simple, user-fiendly lisp command. The simple
combination of el-get and package.el is much of the way there today.

> CB> If the user has to go through a set of steps to install and activate a
> CB> package, how isn't that what ELPA does?
> 
> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).

> [...] from my perspective, IMHO a loader should work
> like package.el.  IOW, whenever and however package.el is loaded
> currently, this loader we're discussing should also be loaded, since
> it's effectively a package manager but with snippets instead of
> packages.

I think that you're conflating package.el with the (still developing)
policy concerning the management of the `official' central archive for
it once it's included by default. You say that you want something that
`should work like package.el', but I don't see where you say what your
problem with package.el is except to imply that you think some sort of
overhead will be too high.

I honestly doubt that there is a lot more effort required to create
something that works with package.el than there is to make a snippet
that works automatically with just a simple mechanical checksum
test. Put another way: I doubt that the overhead is likely to be too
high for anyone who's actually creating something that wants to be
shared automatically.  Put another another way, I think that the
snippet of code that can't be a package or in site-lisp but that
several users want to share and use automatically is really unlikely
to exist.

Maybe you could take a look at package.el start with
http://tromey.com/elpa/faq.html and describe what you think is too
much effort? Package.el will be included with emacs, can be (is
being?) developed to make it more user-friendly, and doesn't have the
security issues of the magic directory.

*Chad

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 7 Mar 2011 09:18:44 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 7, 2011, at 8:24 AM, Ted Zlatanov wrote:
>> Bazaar has a plugins directory; files in it are automatically activated,
>> as an example of a user-level facility like this.  Anyhow, my point is
>> that placing a file in a directory is inherently more modular and
>> convenient to the user than augmenting a single file.  Do you disagree?

CB> My point is that the user must take an explicit step to do either, and
CB> that placing a file in a magic directory and then answering questions
CB> about it (both steps are required) is significantly less convenient
CB> and equally or less modular than running an installer command.

CB> So, yes, I do disagree.  What you call `augmenting a single file' can
CB> be done with a simple, user-fiendly lisp command. The simple
CB> combination of el-get and package.el is much of the way there today.

Again you're talking about package installation.  I want to create
$LOAD_DIR/000-keys.el and have it loaded on startup.  I don't want it
installed, managed, or wrapped.  This mechanism will facilitate
bootstrapping el-get, for instance, but it will also enable users to
manage their configurations better.

CB> I think that you're conflating package.el with the (still developing)
CB> policy concerning the management of the `official' central archive for
CB> it once it's included by default. You say that you want something that
CB> `should work like package.el', but I don't see where you say what your
CB> problem with package.el is except to imply that you think some sort of
CB> overhead will be too high.

I was talking about the mechanics as far as Matt's questions: how it
behaves on --batch, etc.  I was not suggesting it should do what
package.el does in terms of functionality.

I don't have a problem with package.el.  I just don't want to have to
create a package in order to have a modular snippet as I described
above.  Why can't the two coexist?

CB> I honestly doubt that there is a lot more effort required to create
CB> something that works with package.el than there is to make a snippet
CB> that works automatically with just a simple mechanical checksum
CB> test. Put another way: I doubt that the overhead is likely to be too
CB> high for anyone who's actually creating something that wants to be
CB> shared automatically.  Put another another way, I think that the
CB> snippet of code that can't be a package or in site-lisp but that
CB> several users want to share and use automatically is really unlikely
CB> to exist.

Who said anything about sharing?  I want to break down my .emacs for
example; why can't I do it by putting 8 files in a directory instead of
writing (load...) 8 times in my .emacs?  Here's that portion of my
current .emacs:

(load "tzz.emacs.start.el")
(load "tzz.emacs.functions.el")
(load "tzz.emacs.libraries.el")
(load "tzz.emacs.colors.el")
(load "tzz.emacs.options.el")
(load "tzz.emacs.keys.el")
(load "tzz.emacs.custom.el")
(load "tzz.emacs.skeletons.el")

I want, instead, to customize load-dir to ~/emacs.d/conf.d and put those
8 files in it.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chad Brown <yandros@MIT.EDU> writes:
> Put another another way, I think that the
> snippet of code that can't be a package or in site-lisp but that
> several users want to share and use automatically is really unlikely
> to exist.

Ever heard about emacswiki?

What do you think the chances are that authors of snippets and scripts
shared over at emacswiki get to package them to ELPA?  My take is very
very very low, low enough that I wrote el-get just so that I don't have
to take the chance.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 8, 2011, at 9:36 AM, Dimitri Fontaine wrote:

> Chad Brown <yandros@MIT.EDU> writes:
>> Put another another way, I think that the
>> snippet of code that can't be a package or in site-lisp but that
>> several users want to share and use automatically is really unlikely
>> to exist.
> 
> Ever heard about emacswiki?

I think that you haven't been following the conversation closely enough to understand what ``share and use automatically'' means.  The `automatically' part of that sentence is important.

It turns out that what Ted really wants is a way to load all the elisp files in a directory without listing them explicitly. The entire security, package, and downloader discussion (in which I suggested using el-get) was a side-effect of premature overgeneralization.

I hope that helps,
*Chad

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).

There's no reason why package.el can't accept any random elisp file.


        Stefan

Re: user-controlled load-path extension: load-dir

[PJ Weisberg]

On 3/7/11, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> They are not packages, they are snippets of code.  ELPA requires far
>> more structure and many more steps.  For what I'm proposing, a lot less
>> work is required (just a y/n prompt the first time a snippet is found to
>> ensure it's not placed there maliciously).
>
> There's no reason why package.el can't accept any random elisp file.

I think this proposal is really about code snippets that people would
otherwise just cut and paste into their .emacs file.  The average
user's .emacs often winds up containing mostly code they found
somewhere and use without really understanding it.  Dropping each
snippit in its own file would be a big help if the user ever did need
to debug some problem with his init, or if he decided one day to
actually learn elisp.

(I din't know if the security/hash thing is really necessary.  If the
attacker already has write access to your home directory, how much
worse can it get?)

-PJ

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> I think this proposal is really about code snippets that people would
> otherwise just cut and paste into their .emacs file.  The average
> user's .emacs often winds up containing mostly code they found
> somewhere and use without really understanding it.  Dropping each
> snippit in its own file would be a big help if the user ever did need
> to debug some problem with his init, or if he decided one day to
> actually learn elisp.

I'm far from convinced it's better for people to drop random chunks of
configuration into separate files rather than all in the .emacs file.

If you're talking about downloading random files containing
configuration code (so the user doesn't have to create a file, inventing
a file name for it), then maybe that could make sense, but I haven't
seen such files very much, and I suspect many of them could easily be
turned either into themes or into packages installable via package.el.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 07 Mar 2011 21:46:10 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I think this proposal is really about code snippets that people would
>> otherwise just cut and paste into their .emacs file.  The average
>> user's .emacs often winds up containing mostly code they found
>> somewhere and use without really understanding it.  Dropping each
>> snippit in its own file would be a big help if the user ever did need
>> to debug some problem with his init, or if he decided one day to
>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

It's not "better," it's what people already do.  Emacs would just make
it easier than the current situation, which is to drop a file plus edit
the .emacs every time a file is added or deleted.  What other choice is
there to modularize .emacs?

SM> If you're talking about downloading random files containing
SM> configuration code (so the user doesn't have to create a file, inventing
SM> a file name for it), then maybe that could make sense, but I haven't
SM> seen such files very much, and I suspect many of them could easily be
SM> turned either into themes or into packages installable via package.el.

Obviously that works for package managers like el-get.

OK, let's say I have the 8 files I listed earlier (tzz.emacs.*.el).  I
want to load them modularly.  So I put them in the load-dir.  Do I have
to make 8 packages?  And every time I update one of those files, do I
have to repackage it with a new version?  That seems workable but
tedious compared to the code proposed by Ben Key and Evans Winner.

If you're against including the load-dir in the core and enabling it by
default, how about making it a GNU ELPA package so it's easily
installable?

On Tue, 8 Mar 2011 01:14:37 -0600 Ben Key <bkey76@gmail.com> wrote: 

BK> I am attaching to this message a quick implementation of this feature I
BK> threw together in about an hour.  I am not familiar with the Emacs package
BK> mechanism but I am certain that this could be converted to an Emacs package
BK> without too much work thus making it easier to install.

On Tue, 08 Mar 2011 02:58:03 -0700 Evans Winner <ego111@gmail.com> wrote: 

EW> For what it's worth, this is the thing I have in my .emacs.
EW> It does what I want it to, which is just to allow me to
EW> break my .emacs into various not-excessively-long files
EW> organized by subject, like email config, org-mode config and
EW> so-forth.

Thanks for your code.  Both of your solutions are similar to what I
would like to use, but Mike Mattie's concerns about startup behavior
(mainly for daemon mode, for --batch mode, and when there are errors)
are valid.  The recursion behavior should probably not be the default by
parallel with other such systems I listed, e.g. /etc/rc.d and Bazaar's
plugins directory.  It's easy enough to make it optional or to write a
snippet that loads a specific subdirectory.  The argument I have against
recursion is that it's hard to tell at a glance what's going to get
loaded.

It seems no one thinks prompting about new or changed files in the
load-dir is worthwhile, so never mind about that part of my proposal.

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?

It's a bit too trivial to be a package.

Just to be specific: are you simply asking for a function to iterates
through the files in a directory loading them?  Or is it more
complicated than that?

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 08 Mar 2011 12:14:52 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you're against including the load-dir in the core and enabling it by
>> default, how about making it a GNU ELPA package so it's easily
>> installable?

CY> It's a bit too trivial to be a package.

CY> Just to be specific: are you simply asking for a function to iterates
CY> through the files in a directory loading them?  Or is it more
CY> complicated than that?

I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
when set to a list of directories, will tell Emacs to load all the files
in the directory without recursion.  The default should be nil.  This
should not require .emacs modifications.

I suggested packaging it as an alternative in case it was not allowed in
the Emacs core.  I realize it seems trivial but it's really convenient
for managing snippets, ELisp files, and modularizing .emacs.

On Tue, 8 Mar 2011 10:30:14 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> It turns out that what Ted really wants is a way to load all the elisp
CB> files in a directory without listing them explicitly. The entire
CB> security, package, and downloader discussion (in which I suggested
CB> using el-get) was a side-effect of premature overgeneralization.

If this makes it into Emacs, the effect for Dimitri's el-get will be to
simplify the installer and make it Just Work, without modifying the
load-path.  In fact installing any ELisp .el file *manually* will be
simpler.  Helping el-get was the reason I started this thread, but I
have selfish reasons too: it would make my life much easier.

el-get may choose to use this facility directly as well (to manage
EmacsWiki recipes, for instance), but I don't know Dimitri's plans and
it already manages recipes in subdirectories.

Ted

Re: user-controlled load-path extension: load-dir

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Tue, Mar 08 2011, Ted Zlatanov wrote:

> I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
> when set to a list of directories, will tell Emacs to load all the files
> in the directory without recursion.  The default should be nil.  This
> should not require .emacs modifications.

Hum, I'd like to have that defcustom (or another one) to have a
directory where Emacs would not stupidly load the file, but build an
autoload file for them and load that one.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>> I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
>> when set to a list of directories, will tell Emacs to load all the files
>> in the directory without recursion.  The default should be nil.  This
>> should not require .emacs modifications.

> Hum, I'd like to have that defcustom (or another one) to have a
> directory where Emacs would not stupidly load the file, but build an
> autoload file for them and load that one.

That would make more sense, indeed.  That's also closer to what
package.el (w|sh)ould do with random .el files.


        Stefan

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Hum, I'd like to have that defcustom (or another one) to have a
>> directory where Emacs would not stupidly load the file, but build an
>> autoload file for them and load that one.
>
> That would make more sense, indeed.  That's also closer to what
> package.el (w|sh)ould do with random .el files.

If said random .el file follows the Emacs Lisp file conventions, and has
a `;; Version: XX.XX.XX' line in its header section, it's already a
valid package.  M-x package-install-file will do the right thing.

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>>> I think this proposal is really about code snippets that people would
>>> otherwise just cut and paste into their .emacs file.  The average
>>> user's .emacs often winds up containing mostly code they found
>>> somewhere and use without really understanding it.  Dropping each
>>> snippit in its own file would be a big help if the user ever did need
>>> to debug some problem with his init, or if he decided one day to
>>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

> It's not "better," it's what people already do.

I must be missing something.  I see two different things on the web:
packages on the one hand and Elisp configuration chunks on the other.
The first come in files you can download, the other comes in
<pre>...</pre> elements in HTML pages which you can copy&paste.
My claim is that the first can be better handled by package.el or themes
and the other works just fine in .emacs.

> Emacs would just make it easier than the current situation, which is
> to drop a file plus edit the .emacs every time a file is added or
> deleted.

If you drop the file with package.el you don't need to edit the
.emacs correspondingly.

> want to load them modularly.  So I put them in the load-dir.  Do I have
> to make 8 packages?  And every time I update one of those files, do I
> have to repackage it with a new version?  That seems workable but
> tedious compared to the code proposed by Ben Key and Evans Winner.

If you're really talking about configuration code rather than packages,
then I tend to assume that people whose .emacs is so large as to need
modularization can figure out which kind of modularization they want and
implement (or copy&paste) the corresponding loop to load the various files.

I'm not even sure why you'd want to "modularize" in this way: my .emacs
was fairly large but splitting it into separate files never seemed like
a good way to help, since I'd then have to figure out how to make C-s
and M-/ find matches in neighboring files.  Instead I "split" it with
outline-minor-mode.

> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?

Such a package would be perfectly acceptable, yes.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> I think this proposal is really about code snippets that people would
>>>> otherwise just cut and paste into their .emacs file.  The average
>>>> user's .emacs often winds up containing mostly code they found
>>>> somewhere and use without really understanding it.  Dropping each
>>>> snippit in its own file would be a big help if the user ever did need
>>>> to debug some problem with his init, or if he decided one day to
>>>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

>> It's not "better," it's what people already do.

SM> I must be missing something.  I see two different things on the web:
SM> packages on the one hand and Elisp configuration chunks on the other.
SM> The first come in files you can download, the other comes in
SM> <pre>...</pre> elements in HTML pages which you can copy&paste.
SM> My claim is that the first can be better handled by package.el or themes
SM> and the other works just fine in .emacs.

There is a third class: streams (or snippets) of code I don't want to
keep inside my .emacs because I don't like to make it big.  There are
some users who will use and enjoy this functionality, even if other
users won't.  As I proposed it, it will be off by default so it only
benefits those who enable it and won't harm those who don't.

This kind of modularization doesn't make it to the web as much because
it's usually specific to the user's needs.  Here are some examples:

http://www.emacswiki.org/emacs/JoelAdamson#toc5
http://www.io.com/~jimm/emacs_tips.html#my-dot-emacs
http://www.emacswiki.org/emacs/DotEmacsModular
https://github.com/technomancy/emacs-starter-kit
http://www.emacswiki.org/emacs/InitSplit
http://stackoverflow.com/questions/2079095/how-to-modularize-an-emacs-configuration

It's a common need as you can see.

In the Emacs core it will be easier to enable it, that's all.  Otherwise
the instructions have to be "install package load-dir, THEN do the rest"
which is a bit more work, requires a network connection, etc.

>> want to load them modularly.  So I put them in the load-dir.  Do I have
>> to make 8 packages?  And every time I update one of those files, do I
>> have to repackage it with a new version?  That seems workable but
>> tedious compared to the code proposed by Ben Key and Evans Winner.

SM> If you're really talking about configuration code rather than packages,
SM> then I tend to assume that people whose .emacs is so large as to need
SM> modularization can figure out which kind of modularization they want and
SM> implement (or copy&paste) the corresponding loop to load the various files.

Yes, but you're assuming managing configuration modules in monolithic
Emacs Lisp is the best way.  Give us something simple and easy to manage
the loop at the filesystem level, so we don't have to write it
ourselves.

SM> I'm not even sure why you'd want to "modularize" in this way: my .emacs
SM> was fairly large but splitting it into separate files never seemed like
SM> a good way to help, since I'd then have to figure out how to make C-s
SM> and M-/ find matches in neighboring files.  Instead I "split" it with
SM> outline-minor-mode.

For me it works better.  I like small files; outline-minor-mode and
folding-mode don't work for me.  I suspect I'm not the only one.  See
the URLs above for a list of similar needs.

>> If you're against including the load-dir in the core and enabling it by
>> default, how about making it a GNU ELPA package so it's easily
>> installable?

SM> Such a package would be perfectly acceptable, yes.

I'll work with Ben Key to do this in a package if you and Chong Yidong
reject this functionality from the Emacs core.

Ted

Re: user-controlled load-path extension: load-dir

[Jan D.]

Ted Zlatanov skrev 2011-03-08 22:38:
> On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier<monnier@IRO.UMontreal.CA>  wrote:
>
> SM>  If you're really talking about configuration code rather than packages,
> SM>  then I tend to assume that people whose .emacs is so large as to need
> SM>  modularization can figure out which kind of modularization they want and
> SM>  implement (or copy&paste) the corresponding loop to load the various files.
>
> Yes, but you're assuming managing configuration modules in monolithic
> Emacs Lisp is the best way.  Give us something simple and easy to manage
> the loop at the filesystem level, so we don't have to write it
> ourselves.
>
> SM>  I'm not even sure why you'd want to "modularize" in this way: my .emacs
> SM>  was fairly large but splitting it into separate files never seemed like
> SM>  a good way to help, since I'd then have to figure out how to make C-s
> SM>  and M-/ find matches in neighboring files.  Instead I "split" it with
> SM>  outline-minor-mode.
>
> For me it works better.  I like small files; outline-minor-mode and
> folding-mode don't work for me.  I suspect I'm not the only one.  See
> the URLs above for a list of similar needs.
>

As others have said, I was also suprised that dropping a .el-fil in 
~/emacs.d didn't load it by default.  My emacs is quite big and I'd love 
to split it up.  Another thing I really would like is for customize to 
save its stuff in its own file and not in my .emacs.  With a load-dir 
this would be trivial.

There is a real need an packages isn't it.  Its like wanting a rowing 
boat and told to use the latest battle ship instead.

	Jan D.

Re: user-controlled load-path extension: load-dir

[Christoph Scholtes]

On 3/9/2011 12:06 AM, Jan D. wrote:

> Another thing I really would like is for customize to
> save its stuff in its own file and not in my .emacs. With a load-dir
> this would be trivial.

How about this:

(setq custom-file "~/.emacs.d/init-custom.el")
(load custom-file)

See documentation (C-h v custom-file) for details.

Christoph

Re: user-controlled load-path extension: load-dir

[Jan Djärv]

Christoph Scholtes skrev 2011-03-09 08.17:
> On 3/9/2011 12:06 AM, Jan D. wrote:
>
>> Another thing I really would like is for customize to
>> save its stuff in its own file and not in my .emacs. With a load-dir
>> this would be trivial.
>
> How about this:
>
> (setq custom-file "~/.emacs.d/init-custom.el")
> (load custom-file)
>

Yes, but this is missing the point.  I can make an load-dir feature in .emacs 
too.  It is having this in emacs core, working without extra user work that is 
the point.

	Jan D.

`custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Drew Adams]

> >> Another thing I really would like is for customize to
> >> save its stuff in its own file and not in my .emacs. With 
> >> a load-dir this would be trivial.
> >
> > How about this:
> > (setq custom-file "~/.emacs.d/init-custom.el")
> > (load custom-file)
> 
> It is having this in emacs core, working without extra 
> user work that is the point.

My impression is that many Emacs (most?) users are unaware of `custom-file'.
That's too bad, IMO.

I think Emacs should use a separate custom file _by default_ (empty initially),
and thus, by default, keep separate (a) the direct user editing and (b) the
automatic modifications done by Customize.

I would be in favor of having `custom-file' default to ~/.emacs-custom.el or
~/.emacs.d/user-custom.el or something.

By itself, that wouldn't clue users in.  A separate question would be whether to
also have an (almost) empty `.emacs' by default, containing just an explicit
load of `custom-file'.  I'd say yes.

Even if we did not provide a (nearly empty) init file and an (empty)
`custom-file' by default, documenting the use of `custom-file' at the same place
where we document `.emacs' (node `Init File') would be helpful, IMO.  Even just
adding a cross reference there plus a brief mention of `custom-file' would be
helpful.  Today, the `custom-file' doc is buried in the Customization section,
with no mention in the init-file section.

If users did have a default init file and `custom-file' (the former loading the
latter, the latter being empty), then they would more easily keep separate their
own Lisp customizations from any customizations made by Customize.  Mixing the
two is error prone (hence the warning comment we insert today).

Users would be free to add their own Lisp customizations before or after the
load of `custom-file'.  They would also be free to define `custom-file'
conditionally in the init file.  And they could remove the load of `custom-file'
altogether or move it to some other file that gets loaded (i.e. move the load
away from the top level).

IOW, they could do everything they do today.  The only changes would be that by
_default_:

1. Users would have an init file (~/.emacs).

2. The init file would just load `custom-file' (explicitly).  Otherwise it would
be empty.

3. `custom-file' would default to ~/.emacs-custom.el or
~/.emacs.d/user-custom.el or similar (TBD).

Except for #3, I've suggested something similar before (thread "always put
Customizations in `custom-file', never in `user-init-file',
http://lists.gnu.org/archive/html/emacs-devel/2007-12/msg00567.html).

At that time, some were in favor, a few were opposed.  Richard turned it down in
favor of fixing bugs for Emacs 22.  XEmacs apparently already has something like
this (see the thread).

A variant of what's proposed above would be to not have a default init file with
an explicit load of `custom-file', but to just load `custom-file' automatically
after the init file if it is not loaded before then.  This was discussed in the
thread as well.  That too would be OK by me.  (See the thread for details.)

The discussion in that thread also dealt with the possibility of automigrating
existing Customize customizations from .emacs to `custom-file'.  Leaving out
migration altogether - or at least making it optional (on-demand) - would
perhaps lead us to an easier consensus about using `custom-file' by default. 

Perhaps this topic is worth revisiting?

`custom-file' and init-file

[Stephen J. Turnbull]

Drew Adams writes:

 > I would be in favor of having `custom-file' default to
 > ~/.emacs-custom.el or ~/.emacs.d/user-custom.el or something.

FWIW, XEmacs uses the following scheme.

(1) The startup code automatically loads custom-file, if non-nil and
    it exists, *after* loading the init file, if that exists.
    (custom-file will be loaded if init file doesn't exist.)
(2) custom-file defaults to (the equivalent of) .emacs.d/custom.el.

Note that the custom file does not need to be on the load-path since
it has its own location variable, so the name conflict doesn't cause a
problem.  I've not heard of a real issue about the name in 15 years of
use of this scheme.  My own experience is that it's only an issue if
I'm working on lisp/custom.el, in which case it's often prompted by
some problem with ~/.xemacs/custom.el, and then the buffer name
conflict is mildly annoying.

 > By itself, that wouldn't clue users in.  A separate question would
 > be whether to also have an (almost) empty `.emacs' by default,
 > containing just an explicit load of `custom-file'.  I'd say yes.

Note that having a explicit load would require either a conditional or
that custom-file exist.  The XEmacs scheme makes this set of issues
moot.

The only real problem with the XEmacs scheme is that some people have
init files that depend on the custom file.  In that case the remedy is
somewhat ugly but effective.  Add

(load custom-file)
(add-hook 'after-init-hook `(lambda () (setq custom-file ,custom-file)))
(setq custom-file nil)

early enough.

Re: `custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Evans Winner]

,------ Drew Adams wrote ------

|    By itself, that wouldn't clue users in.  A separate
|    question would be whether to also have an (almost)
|    empty `.emacs' by default, containing just an explicit
|    load of `custom-file'.  I'd say yes.

Out of curiosity, would this complicate the process of Emacs
installation?  Is the idea that the first time a user runs
Emacs, it checks to see if there is a .emacs in (getenv
"HOME"), and if not, creates it?  Would that complicate
things for system administrators?  Presumably they should
use site-init.el, but I know some add some kind of simple
.emacs file to the user account creation routine.

By the way, for the default location of the custom file, I
would suggest ~/.emacs.d.  Right now my ~/.emacs.d is filled
with things put there by elpa, predictive, org-mode, rcirc,
my bookmarks file is there.  I used to put my own one-off
third-party libraries there, but I gave up and now put them
in ~/emacs.  I also put my own hacks in ~/.emacs-local and
use the `load-dir' function I posted to load all that.
Anyway, the convention seems to be that things
auto-generated by Emacs and such things belong in
~/.emacs.d.

|    Users would be free to add their own Lisp
|    customizations before or after the load of
|    `custom-file'.  

I think this is important.  I only use the customize
interface, really, when I don't understand something well
enough to handle it in my .emacs.  Actually, there's not
much left in my custom file now, because I tend to move
things out of it to my own init routines as soon as I do
understand something.  I like to feel like I know what is
going on, and like I can change things just by typing in a
buffer, instead of fiddling with the customize interface.
The result, and I can't be alone in this, is that I make the
call to load the custom file at the beginning of my .emacs,
so that if I happen to have defined something twice by
mistake (or because I am too lazy to fix it) the result will
be whatever I have defined in my .emacs.

RE: `custom-file' and init-file [was: user-controlled load-pathextension: load-dir]

[Drew Adams]

> |    By itself, that wouldn't clue users in.  A separate
> |    question would be whether to also have an (almost)
> |    empty `.emacs' by default, containing just an explicit
> |    load of `custom-file'.  I'd say yes.
> 
> Out of curiosity, would this complicate the process of Emacs
> installation?

Dunno.  Don't think so.

> Is the idea that the first time a user runs
> Emacs, it checks to see if there is a .emacs in (getenv
> "HOME"), and if not, creates it?

Yes.  That is one possibility.
(Of course, emacs -Q would act as now: ignore any init file.)

> Would that complicate things for system administrators?

Dunno.  Why would it?

> Presumably they should
> use site-init.el, but I know some add some kind of simple
> .emacs file to the user account creation routine.

Well if the sysadmins give their users a standard (for their organization)
default .emacs, then that would be used, no?  IOW, presumably they or the user
would put that file in the user's $HOME, so it would be picked up by Emacs -
same as now.  I don't see anything changing here.

> By the way, for the default location of the custom file, I
> would suggest ~/.emacs.d.

Sounds good to me.

But see also the other possibility discussed in the referenced thread and
mentioned again today by Stephen: we autoload `custom-file' after the .emacs,
etc.  Either way is fine with me, and there are no doubt other possibilities.

The basic idea is to give the user a `custom-file' by default (empty), and have
Emacs automatically load it.  Users would still have control over whether and
when to load it.

Customizations by Customize would always go to the `custom-file', unless
`custom-file' raises an error.  (Or is some particular value - see the thread
for other possibilities here.)  The point is that we would make sure that users
would still have the possibility of not using `custom-file' at all and thus
having Customize write to .emacs instead. That just would not be the default
behavior.

Re: `custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Stephen J. Turnbull]

Evans Winner writes:

 > Out of curiosity, would this complicate the process of Emacs
 > installation?

That depends on whether the recommendation to use a separate
custom-file is strong enough to put in an automatic "migration"
process.  XEmacs did that; many users went to the trouble of
acknowledging its convenience, but other users suffered data loss
while converting .emacs to XEmacs conventions.  That said, such data
loss is rather unlikely for Emacs users (the problem occurs if the
init file signals while loading; this is unlikely in Emacs -- few
users mess with installing a new Emacs while their .emacs is
non-working! -- but is somewhat likely if you load an .emacs designed
for Emacs into XEmacs).

 > Is the idea that the first time a user runs
 > Emacs, it checks to see if there is a .emacs in (getenv
 > "HOME"), and if not, creates it?

In Drew's scheme, that would probably be the way to go.  (This is very
safe compared to "migrating" an existing .emacs.)  In the XEmacs
scheme (which normally loads custom-file after the init file), it's
unnecessary.

 > Would that complicate things for system administrators?

Slightly.  Probably not a big deal though.  The .emacs file is
explicitly for users to customize, so system administrators should not
count on it being present or on it containing particular code.  That's
what the site init and default files are for.

 > The result, and I can't be alone in this, is that I make the
 > call to load the custom file at the beginning of my .emacs,
 > so that if I happen to have defined something twice by
 > mistake (or because I am too lazy to fix it) the result will
 > be whatever I have defined in my .emacs.

This requires a bit of extra one-time effort in the XEmacs scheme
(which implicitly loads custom-file after the init file).  We consider
it worth it because of the convenience and transparency of the process
for almost all users.

Since the snippet used is entirely generic, it could be wrapped up in
a function `load-custom-file-now' or similar.

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Jan Djärv writes:

 > Christoph Scholtes skrev 2011-03-09 08.17:
 > > On 3/9/2011 12:06 AM, Jan D. wrote:
 > >
 > >> Another thing I really would like is for customize to
 > >> save its stuff in its own file and not in my .emacs. With a load-dir
 > >> this would be trivial.

 > > How about this:

 > > (setq custom-file "~/.emacs.d/init-custom.el")
 > > (load custom-file)

 > Yes, but this is missing the point.  I can make an load-dir feature
 > in .emacs too.  It is having this in emacs core, working without
 > extra user work that is the point.

Yeah, except that as proposed you'll have to enable it in the init
file anyway since it's off by default.

It's also not entirely trivial, as experience in XEmacs shows that the
decision about whether to load such things (specifically custom-file)
*before* or *after* the user init file is a delicate one.  I can only
imagine that a load-dir would very likely be the source of numerous
bugs as some snippets conflict with or depend on others but the
automatic loader gets them in the wrong order.

Note also that .d directories generally use some convention (such as
file names starting with fixed width integers) that ensure that
snippets sort in the right order.  More modern systems use (please sit
down, you're in for a shock) full-blown dependency systems (requires,
provides, conflicts, etc) in the snippets, which avoids the need to
maintain explicit order in favor of a partial order.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> I can only imagine that a load-dir would very likely be the source
SJT> of numerous bugs as some snippets conflict with or depend on others
SJT> but the automatic loader gets them in the wrong order.

We're just adding a mechanism for easy personal code deployment and
modularization, not a cure for broken code.  If there's a load order
conflict the user will have to figure it out, just like putting loads in
the wrong order in the .emacs file.

SJT> Note also that .d directories generally use some convention (such as
SJT> file names starting with fixed width integers) that ensure that
SJT> snippets sort in the right order.

I'm sure users can use that (I will).  But they can also specify the
order with explicit loads if that's what they need.

SJT> More modern systems use (please sit down, you're in for a shock)
SJT> full-blown dependency systems (requires, provides, conflicts, etc)
SJT> in the snippets, which avoids the need to maintain explicit order
SJT> in favor of a partial order.

That's, again, overengineering the problem (as Mark Twain put it, "with
all the modern inconveniences").

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
 > 
 > SJT> I can only imagine that a load-dir would very likely be the source
 > SJT> of numerous bugs as some snippets conflict with or depend on others
 > SJT> but the automatic loader gets them in the wrong order.
 > 
 > We're just adding a mechanism for easy personal code deployment and
 > modularization, not a cure for broken code.

There's no broken code here, just code not engineered with automatic
loading in mind.  Ie, it's the idea of automatically loading random
snippets that is broken: it's a mechanism for speeding up the mess, as
Robert Townsend put it.  I don't think that belongs in core, but
rather should be an individually maintained function, tuned to each
person's requirements.

As for easy personal code deployment and modularization, there's
nothing that says that deployment and modularization requires
implementation as separate files, and in fact using separate files for
most things that you would do in an init file is an annoyance as
Stefan points out.  Stefan's recommendation of using outline minor
mode easily scales to a 100kB file for me in other contexts, I imagine
it would work fine for the init file.  While I admit YMMV as a user,
from the point of view of Emacs maintainers this blows away the
load-dir idea in terms of maintainability of Emacs core.

Note that the good reasons for using separate files (assuming you use
an editor that supports things like outline minor mode -- you do,
don't you?<wink>) are not really compatible with automatic
installation and loading.  They're things like lazy loading and
optional loading to avoid memory bloat and namespace pollution.  Lazy
loading could be a win (somebody mentioned autoloading instead of
automatic loading at init time), therefore, but guess what? that
requires reading code and deciding where to put autoload cookies.

 > SJT> Note also that .d directories generally use some convention (such as
 > SJT> file names starting with fixed width integers) that ensure that
 > SJT> snippets sort in the right order.
 > 
 > I'm sure users can use that (I will).  But they can also specify the
 > order with explicit loads if that's what they need.

Neither of those is easy or automatic.  To the extent that such are
needed, people are going to have to sit down and read code to figure
out what the problems are.  That's precisely what the system is
supposed to avoid.

Note that you also have a problem that such explicitly loaded files
will often need to be moved out of the load-dir, as they won't be
designed to be idempotent.  Mostly not a problem, of course, and
therefore all the more annoying and confusing when it does arise.
 
 > SJT> More modern systems use (please sit down, you're in for a shock)
 > SJT> full-blown dependency systems (requires, provides, conflicts, etc)
 > SJT> in the snippets, which avoids the need to maintain explicit order
 > SJT> in favor of a partial order.
 > 
 > That's, again, overengineering the problem (as Mark Twain put it, "with
 > all the modern inconveniences").

Well, no, it's not.  The problem in question is maintaining a .d
directory without requiring the user to have a clue.  If Stefan and
Yidong could get away with "Ted Z says that's overengineering so we
ain't gonna touch it", it might be worth putting this in core.  But my
recommendation to them is "snippet conflicts are going to be a
persistent annoyance in the long term; let Ted Z and Jan D maintain a
package and deal with the PEBKACs and RFEs."

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 04:33:16 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
>> 
SJT> I can only imagine that a load-dir would very likely be the source
SJT> of numerous bugs as some snippets conflict with or depend on others
SJT> but the automatic loader gets them in the wrong order.
>> 
>> We're just adding a mechanism for easy personal code deployment and
>> modularization, not a cure for broken code.

SJT> There's no broken code here, just code not engineered with automatic
SJT> loading in mind.

By your logic out-of-order startup scripts in /etc/rc.d should be
managed with an external system because simple shell scripts are not
engineered with automatic loading in mind.  Sometimes things really are
independent and you have to trust the user.

SJT> Ie, it's the idea of automatically loading random snippets that is
SJT> broken: it's a mechanism for speeding up the mess, as Robert
SJT> Townsend put it.  I don't think that belongs in core, but rather
SJT> should be an individually maintained function, tuned to each
SJT> person's requirements.

OK, thank you for your opinion.

SJT> As for easy personal code deployment and modularization, there's
SJT> nothing that says that deployment and modularization requires
SJT> implementation as separate files, and in fact using separate files for
SJT> most things that you would do in an init file is an annoyance as
SJT> Stefan points out.  Stefan's recommendation of using outline minor
SJT> mode easily scales to a 100kB file for me in other contexts, I imagine
SJT> it would work fine for the init file.  While I admit YMMV as a user,
SJT> from the point of view of Emacs maintainers this blows away the
SJT> load-dir idea in terms of maintainability of Emacs core.

OK.  What reasons do you have to NOT include the load-dir functionality
in the core if it's disabled by default and must be consciously enabled?
That you're happy without it?

SJT> Note that the good reasons for using separate files (assuming you use
SJT> an editor that supports things like outline minor mode -- you do,
SJT> don't you?<wink>) are not really compatible with automatic
SJT> installation and loading.  They're things like lazy loading and
SJT> optional loading to avoid memory bloat and namespace pollution.  Lazy
SJT> loading could be a win (somebody mentioned autoloading instead of
SJT> automatic loading at init time), therefore, but guess what? that
SJT> requires reading code and deciding where to put autoload cookies.

For me those reasons are modularization and easy code deployment on a
personal level.  I think I've been clear on that and wonder how you
determined your reasons were the good ones.

SJT> Note also that .d directories generally use some convention (such as
SJT> file names starting with fixed width integers) that ensure that
SJT> snippets sort in the right order.
>> 
>> I'm sure users can use that (I will).  But they can also specify the
>> order with explicit loads if that's what they need.

SJT> Neither of those is easy or automatic.  To the extent that such are
SJT> needed, people are going to have to sit down and read code to figure
SJT> out what the problems are.  That's precisely what the system is
SJT> supposed to avoid.

Who said load-dir is supposed to avoid problems?  As with the earlier
comment about broken code, if you screw up your configuration layout in
any form, you go and fix it.  Emacs should indicate where the error
happened, but beyond that it's a user issue.

SJT> Note that you also have a problem that such explicitly loaded files
SJT> will often need to be moved out of the load-dir, as they won't be
SJT> designed to be idempotent.  Mostly not a problem, of course, and
SJT> therefore all the more annoying and confusing when it does arise.

Sorry, I don't follow.  Can you give a specific example?
 
SJT> More modern systems use (please sit down, you're in for a shock)
SJT> full-blown dependency systems (requires, provides, conflicts, etc)
SJT> in the snippets, which avoids the need to maintain explicit order
SJT> in favor of a partial order.
>> 
>> That's, again, overengineering the problem (as Mark Twain put it, "with
>> all the modern inconveniences").

SJT> Well, no, it's not.  The problem in question is maintaining a .d
SJT> directory without requiring the user to have a clue.

NO IT'S NOT!  Why are you assuming you must treat the user like an idiot
who doesn't know he put a file in a directory and can't figure out load
dependencies?  If the user wants hand-holding, he will use Customize and
package.el and el-get and all the other marvels of the modern age.  I
prefer not to, for my own snippets I will write.

SJT> If Stefan and Yidong could get away with "Ted Z says that's
SJT> overengineering so we ain't gonna touch it", it might be worth
SJT> putting this in core.  But my recommendation to them is "snippet
SJT> conflicts are going to be a persistent annoyance in the long term;
SJT> let Ted Z and Jan D maintain a package and deal with the PEBKACs
SJT> and RFEs."

OK, got it.  Thanks.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SJT> There's no broken code here, just code not engineered with automatic
 > SJT> loading in mind.
 > 
 > By your logic out-of-order startup scripts in /etc/rc.d should be
 > managed with an external system because simple shell scripts are not
 > engineered with automatic loading in mind.

Yes.  That's what all of the GNU/Linux distros actually do.  IIRC,
NetBSD does as well, as does Mac OS X (nee FreeBSD), and *BSD is quite
a bit farther to the "user responsibility" side of the field than
GNU/Linux distros are.  Some use a total order maintained mostly by
hand, some use a partial ordering mechanism based on require/conflict/
provides.  I'm not talking theory, I'm talking a decade of practice.

 > OK.  What reasons do you have to NOT include the load-dir functionality
 > in the core if it's disabled by default and must be consciously enabled?
 > That you're happy without it?

No.  I maintain an Emacs, and I really don't want a feature that I am
sure will cause some users pain, but I intend to disclaim
responsibility for.  I've made that mistake a couple of times, and the
minor convenience it would provide to experienced users like you just
doesn't offset the user pain and maintainer embarrassment.

 > For me those reasons are modularization and easy code deployment on a
 > personal level.  I think I've been clear on that and wonder how you
 > determined your reasons were the good ones.

My definition of "good" is pretty strict here.  That is, "the need is
sufficiently great, and the problem sufficiently difficult for users
to solve on their own that it's appropriate to introduce new
problems."  The use of outline-minor-mode introduces no new problems
so the hurdle is low for it.  Since Emacs doesn't have per-function
autoloads, optional loading and autoloading clear a substantially
higher hurdle.  load-dir, OTOH, is easy to implement, and will be even
easier to install as a package.  "Sorry, you're not tall enough to ride
this ride."

 > Who said load-dir is supposed to avoid problems?  As with the earlier
 > comment about broken code, if you screw up your configuration layout in
 > any form, you go and fix it.  Emacs should indicate where the error
 > happened, but beyond that it's a user issue.

But here *Emacs is screwing up the configuration* by automatically
downloading and installing code, at least according to several
advocates' proposals.  Again, *this is not theory*.  This is based on
more than a decade of experience with such systems in XEmacs.  Users
do *not* view this as "oh, I messed up again".  They view it as "if
Emacs is not going to do it right, it should tell me to do it myself."

 > Sorry, I don't follow [the argument about non-idempotent configs].
 > Can you give a specific example?

Hooks written as lambdas, which will be added several times.

 > SJT> If Stefan and Yidong could get away with "Ted Z says that's
 > SJT> overengineering so we ain't gonna touch it", it might be worth
 > SJT> putting this in core.  But my recommendation to them is "snippet
 > SJT> conflicts are going to be a persistent annoyance in the long term;
 > SJT> let Ted Z and Jan D maintain a package and deal with the PEBKACs
 > SJT> and RFEs."
 > 
 > OK, got it.  Thanks.

You're welcome.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 12:20:53 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> For me those reasons are modularization and easy code deployment on a
>> personal level.  I think I've been clear on that and wonder how you
>> determined your reasons were the good ones.

SJT> My definition of "good" is pretty strict here.  That is, "the need is
SJT> sufficiently great, and the problem sufficiently difficult for users
SJT> to solve on their own that it's appropriate to introduce new
SJT> problems."

There's a lot of "daddy knows best" packed in your definition, isn't
there?  I find your "users will feel pain and the maintainer will suffer
embarassment" emotional arguments unappealing and somewhat patronizing,
personally.

>> Who said load-dir is supposed to avoid problems?  As with the earlier
>> comment about broken code, if you screw up your configuration layout in
>> any form, you go and fix it.  Emacs should indicate where the error
>> happened, but beyond that it's a user issue.

SJT> But here *Emacs is screwing up the configuration* by automatically
SJT> downloading and installing code, at least according to several
SJT> advocates' proposals.

I have not advocated such a thing and am against it; if I implied
otherwise I apologize for the misunderstanding.  el-get's *bootstrap*
may reside in the load-dir, but all the packages it manages won't.

SJT> Again, *this is not theory*.  This is based on more than a decade
SJT> of experience with such systems in XEmacs.  Users do *not* view
SJT> this as "oh, I messed up again".  They view it as "if Emacs is not
SJT> going to do it right, it should tell me to do it myself."

I view it as "I enabled the load-dir feature, maybe I should understand
it."  You keep insisting on holding the user's hand.

FWIW XEmacs rewrote and broke my .emacs without prompting last time I
had to use it to debug a Gnus problem, so I'm pessimistic about the way
it holds the user's hand.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > There's a lot of "daddy knows best" packed in your definition, isn't
 > there?

Of course.  That's what a maintainer does, make these kinds of
decisions about what's best for everybody.  That is the point of view
I'm taking, not trying to evaluate the feature on its own terms.

 > I have not advocated [automatically installing and loading code]
 > and am against it; if I implied otherwise I apologize for the
 > misunderstanding.  el-get's *bootstrap* may reside in the load-dir,
 > but all the packages it manages won't.

From the maintainer's point of view, this is *not about you*.  You
have restrained requirements for the feature, yes.  How do you propose
to impose *your* restraints on *others* who want additional features,
though?

 > SJT> Again, *this is not theory*.  This is based on more than a decade
 > SJT> of experience with such systems in XEmacs.  Users do *not* view
 > SJT> this as "oh, I messed up again".  They view it as "if Emacs is not
 > SJT> going to do it right, it should tell me to do it myself."
 > 
 > I view it as "I enabled the load-dir feature, maybe I should understand
 > it."

Again, how do you propose to get all other users to view it that way?
If this is only for you, you already know how to do it.  Obviously,
you expect this to be used by many people.  It needs to meet their
expectations as well as yours, or it's a bad idea to maintain it in
core.

It is my opinion, based on experience, that requests for more
automation will come up over and over again.  Similarly, requests for
support when code that users throw into the pot conflicts with code
already there will come up over and over again.  I don't think this
feature is worth inducing those requests, that's all.

 > You keep insisting on holding the user's hand.

No.  I'm anticipating what *some* users are going to expect of it,
based on experience with maintaining similar features.  I wouldn't
hold the user's hand in this case, I'd simply insist that the feature
be provided as a package.  Then helping them with their misconceptions
(and statistically speaking, there will be a certain fraction of users
who misunderstand) is my privilege as core maintainer, not my
responsibility.  Putting it in core makes it the other way around, and
that will suck for Stefan and Yidong IMO.

 > FWIW XEmacs rewrote and broke my .emacs without prompting last time I
 > had to use it to debug a Gnus problem, so I'm pessimistic about the way
 > it holds the user's hand.

Bingo!  That's exactly the result of the kind of feature creep I'm
worred about here, that I described as a mistake, which I now regret
not opposing more vehemently at the time, and which I am still not in
a position to unilaterally roll back.

And whatever happened to your "I enabled it, I should understand it"
view?  XEmacs was supposed to be a drop-in replacement for Emacs; you
shouldn't be surprised that it uses .emacs as its init file and its
custom-file if it finds it and custom-file isn't explicitly set, just
like Emacs does.  If you had spent the time to think about the
implications of what you were doing, I should think that "xemacs -q"[1]
would be the obvious thing to use to debug Gnus.  After all, the
person who reported the bug certainly wasn't using *your* .emacs in
*her* XEmacs.

But that's not the way you think about it, clearly.  And guess what?
You're right not to think about it that way, just as the users who
won't think so carefully about a trivial feature like load-dir are
right for them.  Life is too short.


Footnotes: 
[1]  Which causes both user-init-file and custom-file to be nil, so no
accesses, let alone writes, to .emacs would occur.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 16:08:15 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> I have not advocated [automatically installing and loading code]
>> and am against it; if I implied otherwise I apologize for the
>> misunderstanding.  el-get's *bootstrap* may reside in the load-dir,
>> but all the packages it manages won't.

SJT> From the maintainer's point of view, this is *not about you*.  You
SJT> have restrained requirements for the feature, yes.  How do you propose
SJT> to impose *your* restraints on *others* who want additional features,
SJT> though?

By making the feature an optional, off-by-default loader and nothing more.

>> I view it as "I enabled the load-dir feature, maybe I should understand
>> it."

SJT> Again, how do you propose to get all other users to view it that way?
SJT> If this is only for you, you already know how to do it.  Obviously,
SJT> you expect this to be used by many people.  It needs to meet their
SJT> expectations as well as yours, or it's a bad idea to maintain it in
SJT> core.

I think you're arguing that the feature will be misinterpreted and
misused.  I disagree and can't find a way to agree with you; I still
think it's simple enough that it won't cause the kind of mismatched
expectations that lead to unwanted behavior and bug reports.

>> You keep insisting on holding the user's hand.

SJT> No.  I'm anticipating what *some* users are going to expect of it,
SJT> based on experience with maintaining similar features.  I wouldn't
SJT> hold the user's hand in this case, I'd simply insist that the feature
SJT> be provided as a package.  Then helping them with their misconceptions
SJT> (and statistically speaking, there will be a certain fraction of users
SJT> who misunderstand) is my privilege as core maintainer, not my
SJT> responsibility.  Putting it in core makes it the other way around, and
SJT> that will suck for Stefan and Yidong IMO.

While I disagree about the consequences you think it will have, I'm
proposing it before committing code exactly so the maintainers can make
up their mind.  I think we've both stated our case clearly at this point.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:
 
 > While I disagree about the consequences you think it will have, I'm
 > proposing it before committing code exactly so the maintainers can make
 > up their mind.  I think we've both stated our case clearly at this point.

Yup.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Wed, Mar 09, 2011 at 08:06:16AM +0100, Jan D. wrote:
> Ted Zlatanov skrev 2011-03-08 22:38:
> > On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier<monnier@IRO.UMontreal.CA>  wrote:
> >
> > SM>  If you're really talking about configuration code rather than packages,
> > SM>  then I tend to assume that people whose .emacs is so large as to need
> > SM>  modularization can figure out which kind of modularization they want and
> > SM>  implement (or copy&paste) the corresponding loop to load the various files.
> >
> > Yes, but you're assuming managing configuration modules in monolithic
> > Emacs Lisp is the best way.  Give us something simple and easy to manage
> > the loop at the filesystem level, so we don't have to write it
> > ourselves.
> >
> > SM>  I'm not even sure why you'd want to "modularize" in this way: my .emacs
> > SM>  was fairly large but splitting it into separate files never seemed like
> > SM>  a good way to help, since I'd then have to figure out how to make C-s
> > SM>  and M-/ find matches in neighboring files.  Instead I "split" it with
> > SM>  outline-minor-mode.
> >
> > For me it works better.  I like small files; outline-minor-mode and
> > folding-mode don't work for me.  I suspect I'm not the only one.  See
> > the URLs above for a list of similar needs.
> >
> 
> As others have said, I was also suprised that dropping a .el-fil in 
> ~/emacs.d didn't load it by default.  My emacs is quite big and I'd love 
> to split it up.  Another thing I really would like is for customize to 
> save its stuff in its own file and not in my .emacs.  With a load-dir 
> this would be trivial.
> 
> There is a real need an packages isn't it.  Its like wanting a rowing 
> boat and told to use the latest battle ship instead.

Can you take a look at Grail http://www.emacswiki.org/emacs/Grail and see if
that meets your needs ? If you want to modularize your configuration it is
explicity designed to do so.

> 
> 	Jan D.
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 0 bytes --]

Re: user-controlled load-path extension: load-dir

[Jan Djärv]

Mike Mattie skrev 2011-03-09 22.57:
> On Wed, Mar 09, 2011 at 08:06:16AM +0100, Jan D. wrote:

>>
>> There is a real need an packages isn't it.  Its like wanting a rowing
>> boat and told to use the latest battle ship instead.
>
> Can you take a look at Grail http://www.emacswiki.org/emacs/Grail and see if
> that meets your needs ? If you want to modularize your configuration it is
> explicity designed to do so.
>

Still not part of the core, and also too complicated.

	Jan D.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 4261 bytes --]

On Tue, Mar 08, 2011 at 04:26:32AM -0600, Ted Zlatanov wrote:
> On Mon, 07 Mar 2011 21:46:10 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> >> I think this proposal is really about code snippets that people would
> >> otherwise just cut and paste into their .emacs file.  The average
> >> user's .emacs often winds up containing mostly code they found
> >> somewhere and use without really understanding it.  Dropping each
> >> snippit in its own file would be a big help if the user ever did need
> >> to debug some problem with his init, or if he decided one day to
> >> actually learn elisp.
> 
> SM> I'm far from convinced it's better for people to drop random chunks of
> SM> configuration into separate files rather than all in the .emacs file.
> 
> It's not "better," it's what people already do.  Emacs would just make
> it easier than the current situation, which is to drop a file plus edit
> the .emacs every time a file is added or deleted.  What other choice is
> there to modularize .emacs?
> 
> SM> If you're talking about downloading random files containing
> SM> configuration code (so the user doesn't have to create a file, inventing
> SM> a file name for it), then maybe that could make sense, but I haven't
> SM> seen such files very much, and I suspect many of them could easily be
> SM> turned either into themes or into packages installable via package.el.
> 
> Obviously that works for package managers like el-get.
> 
> OK, let's say I have the 8 files I listed earlier (tzz.emacs.*.el).  I
> want to load them modularly.  So I put them in the load-dir.  Do I have
> to make 8 packages?  And every time I update one of those files, do I
> have to repackage it with a new version?  That seems workable but
> tedious compared to the code proposed by Ben Key and Evans Winner.
> 
> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?
> 
> On Tue, 8 Mar 2011 01:14:37 -0600 Ben Key <bkey76@gmail.com> wrote: 
> 
> BK> I am attaching to this message a quick implementation of this feature I
> BK> threw together in about an hour.  I am not familiar with the Emacs package
> BK> mechanism but I am certain that this could be converted to an Emacs package
> BK> without too much work thus making it easier to install.
> 
> On Tue, 08 Mar 2011 02:58:03 -0700 Evans Winner <ego111@gmail.com> wrote: 
> 
> EW> For what it's worth, this is the thing I have in my .emacs.
> EW> It does what I want it to, which is just to allow me to
> EW> break my .emacs into various not-excessively-long files
> EW> organized by subject, like email config, org-mode config and
> EW> so-forth.
> 
> Thanks for your code.  Both of your solutions are similar to what I
> would like to use, but Mike Mattie's concerns about startup behavior
> (mainly for daemon mode, for --batch mode, and when there are errors)
> are valid.  The recursion behavior should probably not be the default by
> parallel with other such systems I listed, e.g. /etc/rc.d and Bazaar's
> plugins directory.  It's easy enough to make it optional or to write a
> snippet that loads a specific subdirectory.  The argument I have against
> recursion is that it's hard to tell at a glance what's going to get
> loaded.

I have thought alot about your proposal for automatic loading so that it
is not necessary to edit anything, just drop a file in. There is some
real appeal for that kind of a feature ... but it is hard to implement.

By hard to implement I mean really hard. Just because a file is opened
and written to, it is not necessarily ready to load. I hit C-x s alot
when tweaking files ...

Even if I hacked some kind of inotify support to watch a directory for
IO activity I would never know when to actually load it.

If you can somehow signal when a collection is ready for re-scanning
I can see an implementation ...

> 
> It seems no one thinks prompting about new or changed files in the
> load-dir is worthwhile, so never mind about that part of my proposal.

I don't discount your idea, it's just hard to make a really reliable
implementation IMHO.

> 
> Thanks
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Jan D.]

Mike Mattie skrev 2011-03-09 07:03:

> I have thought alot about your proposal for automatic loading so that it
> is not necessary to edit anything, just drop a file in. There is some
> real appeal for that kind of a feature ... but it is hard to implement.
>
> By hard to implement I mean really hard. Just because a file is opened
> and written to, it is not necessarily ready to load. I hit C-x s alot
> when tweaking files ...
>
> Even if I hacked some kind of inotify support to watch a directory for
> IO activity I would never know when to actually load it.
>
> If you can somehow signal when a collection is ready for re-scanning
> I can see an implementation ...
>

It can't be hard, there are tons of programs that already have this 
feature.  Some have inotify style loading, other just load at startup. 
I would expect load at startup and then with a command (M-x 
load-load-dir or some such).

	Jan D.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Wed, Mar 09, 2011 at 08:20:49AM +0100, Jan D. wrote:
> Mike Mattie skrev 2011-03-09 07:03:
> 
> > I have thought alot about your proposal for automatic loading so that it
> > is not necessary to edit anything, just drop a file in. There is some
> > real appeal for that kind of a feature ... but it is hard to implement.
> >
> > By hard to implement I mean really hard. Just because a file is opened
> > and written to, it is not necessarily ready to load. I hit C-x s alot
> > when tweaking files ...
> >
> > Even if I hacked some kind of inotify support to watch a directory for
> > IO activity I would never know when to actually load it.
> >
> > If you can somehow signal when a collection is ready for re-scanning
> > I can see an implementation ...
> >
> 
> It can't be hard, there are tons of programs that already have this 
> feature.  Some have inotify style loading, other just load at startup. 
> I would expect load at startup and then with a command (M-x 
> load-load-dir or some such).

That I can do.

> 
> 	Jan D.
> 
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 0 bytes --]

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 5562 bytes --]

On Mon, Mar 07, 2011 at 10:24:58AM -0600, Ted Zlatanov wrote:
> On Sat, 5 Mar 2011 11:11:34 -0800 Chad Brown <yandros@MIT.EDU> wrote: 
> 
> CB> On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
> >> 
> >> By analogy consider some of the software that lets you put a snippet in
> >> a conf.d directory, obviously implying that this is convenient for the
> >> user.  This is just a sampling from my machine:
> 
> CB> Those are all machine-wide configurations.  For that, use site-lisp.
> 
> Bazaar has a plugins directory; files in it are automatically activated,
> as an example of a user-level facility like this.  Anyhow, my point is
> that placing a file in a directory is inherently more modular and
> convenient to the user than augmenting a single file.  Do you disagree?
> 
> CB> If the user has to go through a set of steps to install and activate a
> CB> package, how isn't that what ELPA does?
> 
> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).
> 

This whole security thing is a real non-issue. There is no security in
Emacs, it (Emacs) simply uses the underlying system security
model. What is really at issue is whether the user _trusts_ the
code. Trying to address security issues in a software system that has
no security API/model leads to some skewed thinking, and I am not just
splitting hairs. Thinking in terms of trust leads to better reasoning.

> On Sat, 5 Mar 2011 23:21:51 -0800 Mike Mattie <codermattie@gmail.com> wrote: 
> 
> MM> There are ways to solve the problem you are looking at without wiring more
> MM> logic into Emacs. Use the .emacs file as a more sophisticated loader for a
> MM> complex configuration when necessary.
> 
> I can and do, sure.  I don't think it creates a nice user experience.
> 
> MM> I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
> MM> I would look at these solutions first before proposing code that has to
> MM> be wired into the bootstrap codepath.
> 
> Thanks for pointing those out.  Would Grail or tidyconfig work as a core
> option on startup (meaning the user doesn't have to install a package,
> just customize a boolean)?  Do they handle signing the snippets the
> first time they are found?  Can you give us a comparison of them and any
> other similar solutions you know?  They have to be GPL and assigned to
> Emacs to be included.

I doubt Grail would be included in-tree. Maybe I am wrong, but I get that
vibe. I would certianly need to convert some recursive functions to iterative
versions before even bringing Grail to the table. I have been maturing Grail
out-of-tree to ensure the code is robust, designed well, and is targetted at
typical user needs.

> 
> MM> There are a number of issues a reliable loader should address:
> 
> MM> how does it handle errors ?
> 
> MM> how does it handle --batch ?
> 
> MM> how does it handle --daemon ?
> 
> MM> how simple,transparent,debuggable is the loader code ?
> 
> MM> I have not seen any code so far in your proposal unless I accidentally
> MM> skipped or deleted a message with it.
> 
> I can write the code but haven't seen the need for actual code yet (I
> could have just comitted some code to Emacs instead, but find that
> option to be antisocial).
> 
> I'd rather use something that already exists such as Grail or tidyconfig
> that you kindly pointed out.  If you could tell us how any potential
> solutions can help answer the questions above, that would be wonderful.
> I can look at any solutions and evaluate them if you like.

I can only speak for grail, but Grail essentially Handles the
load-path issue. When it comes to load-path Grail is
comprehensive. What grail does not do is automatically load code from
a given directory. Directories that grail looks for are added to
load-path when they exist. Grail looks for files such as keys.el
commands.el, user.el and does load those when found.  They are assumed
to be user created files. When snippets are collected in a directory
such as dist/elisp, or local/elisp, the user is expected to place a
require or similar command in one of these recognized files to
actually load the snippet/whatever at the start.

As far as error-handling goes grail does shine a bit in that errors within
a file are trapped, meaning that mistakes in one file does not cause the
entire configuration to abort leading to a --debug-init situtation. In this
regard Grail is extremely robust.

I will be updating the documentation of Grail extensively over the next couple
few days to describe it's use, usefulness, limitations etc .. I will send you
a mail off-list when I have articulated Grail properly.

> 
> To answer your questions from my perspective, IMHO a loader should work
> like package.el.  IOW, whenever and however package.el is loaded
> currently, this loader we're discussing should also be loaded, since
> it's effectively a package manager but with snippets instead of
> packages.

I am seriously considering how to add the kind of functionality you want to
Grail. I would like to add it to broaden the usefullness of Grail. I do not
like the idea of hacking another load-path 'ish variable into Emacs when simply
adding elisp to manage load-path is an option. Two similar variables is simply
bad design.

> 
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 7 Mar 2011 16:47:27 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> As far as error-handling goes grail does shine a bit in that errors within
MM> a file are trapped, meaning that mistakes in one file does not cause the
MM> entire configuration to abort leading to a --debug-init situtation. In this
MM> regard Grail is extremely robust.

That sounds useful indeed.

MM> I am seriously considering how to add the kind of functionality you want to
MM> Grail. I would like to add it to broaden the usefullness of Grail. I do not
MM> like the idea of hacking another load-path 'ish variable into Emacs when simply
MM> adding elisp to manage load-path is an option. Two similar variables is simply
MM> bad design.

Maybe you could provide separate packages: grail-loader, grail-load-dir
(the snippets directory), and grail-modules (the current Grail modules
like keys.el).  Then users can choose which layout they prefer and still
get the benefit of a safe loader.  Or perhaps grail-load-dir can simply
be part of grail-loader.

Ted

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Tue, Mar 08, 2011 at 04:37:37AM -0600, Ted Zlatanov wrote:
> On Mon, 7 Mar 2011 16:47:27 -0800 Mike Mattie <codermattie@gmail.com> wrote: 
> 
> MM> As far as error-handling goes grail does shine a bit in that errors within
> MM> a file are trapped, meaning that mistakes in one file does not cause the
> MM> entire configuration to abort leading to a --debug-init situtation. In this
> MM> regard Grail is extremely robust.
> 
> That sounds useful indeed.
> 
> MM> I am seriously considering how to add the kind of functionality you want to
> MM> Grail. I would like to add it to broaden the usefullness of Grail. I do not
> MM> like the idea of hacking another load-path 'ish variable into Emacs when simply
> MM> adding elisp to manage load-path is an option. Two similar variables is simply
> MM> bad design.
> 
> Maybe you could provide separate packages: grail-loader, grail-load-dir
> (the snippets directory), and grail-modules (the current Grail modules
> like keys.el).  Then users can choose which layout they prefer and still
> get the benefit of a safe loader.  Or perhaps grail-load-dir can simply
> be part of grail-loader.

I have been reading this thread very carefully and have given the
snippets proposal alot of thought. I am willing to add a snippets
directory like functionality. I would require the user to execute a
command to signal to grail that the user has placed a new file in the
snippets directory.

BTW, Grail does allow you to change the layout, by creating a
grail-cfg.el file in USER_ELISP (see the Grail page for the meaning of
that variable). grail-cfg.el is loaded if it exists right after the
default tree is defined in defvar's and before the heavy lifting is done
in terms of loading.

> 
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 8 Mar 2011 22:26:01 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> I have been reading this thread very carefully and have given the
MM> snippets proposal alot of thought. I am willing to add a snippets
MM> directory like functionality. I would require the user to execute a
MM> command to signal to grail that the user has placed a new file in the
MM> snippets directory.

MM> BTW, Grail does allow you to change the layout, by creating a
MM> grail-cfg.el file in USER_ELISP (see the Grail page for the meaning of
MM> that variable). grail-cfg.el is loaded if it exists right after the
MM> default tree is defined in defvar's and before the heavy lifting is done
MM> in terms of loading.

I think Grail brings too much functionality for the proposal in this
discussion, after looking at its pieces.  Ben Key's implementation is
closer to the simplicity I was hoping for.  That's not to say Grail is
not useful, I just don't think that it is the right solution here.

Ted