user-controlled load-path extension: load-dir

user-controlled load-path extension: load-dir

[Ted Zlatanov]

Sorry if this has been discussed before.  I don't recall such a
discussion and the search terms are too generic.

I think it's useful to load on startup any .el files placed in a
particular directory (if the user approves the directory).  For example,
if the user sets `load-dir' to '("~/.emacs.d/load"),
~/.emacs.d/load/*.el will be loaded on startup.  The `load-path' is not
involved, this would be a straight `eval' call.

`load-dir' is just a name suggestion.

This can be done on the user side, of course, but if Emacs had a
built-in facility to do it, enabling and controlling Emacs
configurations in a modular way would be easier, especially for new
users.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ted Zlatanov <tzz@lifelogs.com> writes:
> I think it's useful to load on startup any .el files placed in a
> particular directory (if the user approves the directory).  For example,
> if the user sets `load-dir' to '("~/.emacs.d/load"),
> ~/.emacs.d/load/*.el will be loaded on startup.  The `load-path' is not
> involved, this would be a straight `eval' call.

The same way we have user-init-file, I too would welcome Emacs to offer
a default user-load-path that's known to be e.g. ~/.emacs.d/load.d.
That would allow third party elisp code to just drop a file in there and
have users all set to use a new facility.

For example, I would use that for el-get and bootstrapping it into the
user's setup would become easier, and would allo to avoid having to edit
user-init-file.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Thu, Mar 03 2011, Dimitri Fontaine wrote:

> The same way we have user-init-file, I too would welcome Emacs to offer
> a default user-load-path that's known to be e.g. ~/.emacs.d/load.d.
> That would allow third party elisp code to just drop a file in there and
> have users all set to use a new facility.

Especially if Emacs was able to build and load an autoload file for the
files placed in that directory. The user would have *nothing* to do to
use a new feature. Not even touching its `user-init-file'.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Julien> Especially if Emacs was able to build and load an autoload file for the
Julien> files placed in that directory. The user would have *nothing* to do to
Julien> use a new feature. Not even touching its `user-init-file'.

package.el does this, more or less.  It doesn't just load all files, but
does some version and dependency checks.  But, e.g., it is trivial for
el-get to be supplied as a package.  This feature is enabled by default
in Emacs 24.

Tom

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 10:18:03 -0700 Tom Tromey <tromey@redhat.com> wrote: 

Julien> Especially if Emacs was able to build and load an autoload file for the
Julien> files placed in that directory. The user would have *nothing* to do to
Julien> use a new feature. Not even touching its `user-init-file'.

Tom> package.el does this, more or less.  It doesn't just load all files, but
Tom> does some version and dependency checks.  But, e.g., it is trivial for
Tom> el-get to be supplied as a package.  This feature is enabled by default
Tom> in Emacs 24.

OK, but there's no general facility to do this.  If I want to bundle up
some code and load it on startup, I have to modify the load-path and my
startup .emacs right now (which encourages monolithic configurations and
other bad practices).  I don't think package.el should be required to do
this kind of modularization.

As far as el-get, I guess it could be made into a package, but it's a
package manager and has to work in cases where package.el is not
available or the user doesn't want to use it.  I don't think el-get is
the only use for the general load-dir facility, in any case.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> package.el does this, more or less.  It doesn't just load all files, but
> does some version and dependency checks.  But, e.g., it is trivial for
> el-get to be supplied as a package.  This feature is enabled by default
> in Emacs 24.

Would it be accepted, though?  If yes, I'd better go begin those FSF
paperwork.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Tom> package.el does this, more or less.  It doesn't just load all files, but
Tom> does some version and dependency checks.  But, e.g., it is trivial for
Tom> el-get to be supplied as a package.  This feature is enabled by default
Tom> in Emacs 24.

Dimitri> Would it be accepted, though?  If yes, I'd better go begin those FSF
Dimitri> paperwork.

Suppose Emacs did add a "load all the files in a directory" feature.
You would tell your users "download this and install it here".
Or you would supply some elisp on a web page for them to eval, and this
elisp would do the downloading.

Well, you can do the exact same things with a package.  You don't need
it to be in elpa.gnu.org.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> Suppose Emacs did add a "load all the files in a directory" feature.
> You would tell your users "download this and install it here".
> Or you would supply some elisp on a web page for them to eval, and this
> elisp would do the downloading.
>
> Well, you can do the exact same things with a package.  You don't need
> it to be in elpa.gnu.org.

What el-get currently does not do is editing any user's file.  What's
proposed in this thread is a way for el-get to bootstrap itself in a
user's setup without having to edit the user's init file.

Now, this would be answered by a known user's default load-path.  Julien
and Ted where adding to it actual loading then autoloads, I would live
without those.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 20:45:45 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

DF> What's proposed in this thread is a way for el-get to bootstrap
DF> itself in a user's setup without having to edit the user's init
DF> file.

s/el-get/any Emacs Lisp code/

The only reason I can think why this would not be acceptable is
security.  Otherwise this makes a lot of sense.

For security I would propose checksumming any code in the load-dir and
when new files are added or existing files are changed, query the user
and store a file checksum in an alist.

I'm willing to make this facility a package that users can install.  But
I think it's better to put it in the Emacs core so it's available by
default and without special action from the user beyond enabling a
single variable.  I feel strongly it would enhance Emacs to provide this.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ted Zlatanov <tzz@lifelogs.com> writes:
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.

My answer there would be to just add to load-path, but have no automatic
mechanism to load the files in there or register the autoloads.  Having
a known place where to download el-get.el and being able to tell user to
just (require 'el-get) is all I want here.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]


On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
> 
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.
> 
> For security I would propose checksumming any code in the load-dir and
> when new files are added or existing files are changed, query the user
> and store a file checksum in an alist.

This would have to happen the first time the user loaded any package, or you have no security. If you do this the first time the user has loaded any package, you have removed the `automatically' part of your proposed system, which is the only real advantage it has.

The only way your proposal seems (to me) to be an improvement is if we train users to ignore the security questions and just always click on `I agree'.  I hope you agree that that isn't a good idea.

Modern users have been trained to expect `Extensions' and `Add-Ons', and don't seem to mind picking from a list (for things like FireFox, a very large list with many overlapping choices). I think that's a better user model to follow.

I hope that helps,
*Chad


[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 4 Mar 2011 12:25:22 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
>> 
>> The only reason I can think why this would not be acceptable is
>> security.  Otherwise this makes a lot of sense.
>> 
>> For security I would propose checksumming any code in the load-dir and
>> when new files are added or existing files are changed, query the user
>> and store a file checksum in an alist.

CB> This would have to happen the first time the user loaded any package,
CB> or you have no security. If you do this the first time the user has
CB> loaded any package, you have removed the `automatically' part of your
CB> proposed system, which is the only real advantage it has.

No.  This query is only for Lisp code placed in the load-dir, not for
packages in general.  So you'd have the query as part of the el-get
installation for instance and it wouldn't show up again.  Compare with
Mac OS X which does this for new applications you've downloaded from the
net, alerting the very first time.

CB> The only way your proposal seems (to me) to be an improvement is if we
CB> train users to ignore the security questions and just always click on
CB> I agree'.  I hope you agree that that isn't a good idea.

It's not a good idea, but it's not what I'm describing either.  This
query should show up very rarely, not all the time.  It's meant for
bootstrapping and for user code.  If more than one file needs to be
approved the user could use `Y' instead of `y', but in any case the
query would be very rare.

CB> Modern users have been trained to expect `Extensions' and `Add-Ons',
CB> and don't seem to mind picking from a list (for things like FireFox, a
CB> very large list with many overlapping choices). I think that's a
CB> better user model to follow.

Let's take an example.  I have 10 pieces of code related to Emacs I want
to load.  Do I have to explicitly list them in my .emacs?  Or can I just
put them in a directory?  The former is twice the work since I have to
put them in some directory anyhow and puts the synchronization burden on
me.  So I can write some Lisp to do the discovery and load them
automatically... which is what I'm offering for everyone's benefit (with
some security features).  I already do this for myself so I can propose
a patch or a package quickly :)

Package management is not what I'm proposing.  The package manager
*installation* would write a bootstrapper piece of code that has to be
approved *once*.  So while the load-dir is useful for installing el-get,
I don't think el-get is the only reason to adopt it.

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 4, 2011, at 12:46 PM, Ted Zlatanov wrote:

> No.  This query is only for Lisp code placed in the load-dir, not for
> packages in general.  So you'd have the query as part of the el-get
> installation for instance and it wouldn't show up again.  Compare with
> Mac OS X which does this for new applications you've downloaded from the
> net, alerting the very first time.

If the user has to go through a set of steps to install and activate a package, how isn't that what ELPA does?

As I'm understanding you, any magic directory is either no less work for the user or has unforgivable security issues.  What am I missing?

*Chad

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]


On Mar 4, 2011, at 11:54 AM, Ted Zlatanov wrote:
> 
> The only reason I can think why this would not be acceptable is
> security.  Otherwise this makes a lot of sense.
> 
> For security I would propose checksumming any code in the load-dir and
> when new files are added or existing files are changed, query the user
> and store a file checksum in an alist.

This would have to happen the first time the user loaded any package, or you have no security. If you do this the first time the user has loaded any package, you have removed the `automatically' part of your proposed system, which is the only real advantage it has.

The only way your proposal seems (to me) to be an improvement is if we train users to ignore the security questions and just always click on `I agree'.  I hope you agree that that isn't a good idea.

Modern users have been trained to expect `Extensions' and `Add-Ons', and don't seem to mind picking from a list (for things like FireFox, a very large list with many overlapping choices). I think that's a better user model to follow.

I hope that helps,
*Chad


[-- Attachment #2: Type: text/html, Size: 1492 bytes --]

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> What el-get currently does not do is editing any user's file.  What's
Dimitri> proposed in this thread is a way for el-get to bootstrap itself in a
Dimitri> user's setup without having to edit the user's init file.

Yes, my proposal accomplishes that.  There are two problems here.

The first problem is, how does the user ever get el-get.el?
Do they download it from a web site?  Do you have some installer elisp
on a web site to eval (this is what I did with ELPA)?  Etc.

Whatever the answer is to that problem, there is a similar
package.el-based answer.

E.g., if you have some elisp on your web site to download and install
el-get.el, just write different elisp (untested):

(let ((buffer (url-retrieve-synchronously "the url/el-get.el")))
  (with-current-buffer buffer
    ;; strip the headers
    (re-search-forward "^$" nil 'move)
    (forward-char)
    (delete-region (point-min) (point))
    (package-install-from-buffer (package-buffer-info) 'single))
  (kill-buffer buffer))

If users just download your file by hand, then they just do that and
then M-x package-install-file.


The second problem is, how does el-get.el get activated?

If el-get.el is an ordinary package, in the package.el sense, then in
Emacs 24 it will just work with zero changes to the user's init file.
Autoloads will be extracted, etc.  You can make it do any
el-get-specific activation quite easily using an autoload comment.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> Whatever the answer is to that problem, there is a similar
> package.el-based answer.
>
> E.g., if you have some elisp on your web site to download and install
> el-get.el, just write different elisp (untested):

Here's the code to copy paste and run to install el-get currently.

(url-retrieve
 "https://github.com/dimitri/el-get/raw/master/el-get-install.el"
 (lambda (s)
   (end-of-buffer)
   (eval-print-last-sexp)))

>     (package-install-from-buffer (package-buffer-info) 'single))

I can see your point about el-get being a package even if not included
in ELPA by default, now that package.el is part of Emacs24.  That leaves
any user of Emacs23 with no solution though.  I will consider that at
some point I guess, but currently el-get allows you to get package.el in
emacs23 :)

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> I can see your point about el-get being a package even if not
Dimitri> included in ELPA by default, now that package.el is part of
Dimitri> Emacs24.  That leaves any user of Emacs23 with no solution
Dimitri> though.  I will consider that at some point I guess, but
Dimitri> currently el-get allows you to get package.el in emacs23 :)

If somebody added the feature you want to Emacs, presumably it would
only be in Emacs 24, leaving you in the same situation.

Or maybe the proposal is to put it into Emacs 23.  But then we might as
well just put package.el into Emacs 23.

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> If somebody added the feature you want to Emacs, presumably it would
> only be in Emacs 24, leaving you in the same situation.

Point.  I would still prefer to depend on user-load-path or something,
it strikes me as more general: you don't have to edit existing scripts.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

Dimitri> Point.  I would still prefer to depend on user-load-path or something,
Dimitri> it strikes me as more general: you don't have to edit existing scripts.

Now you're just looking for reasons to avoid it :) 

Tom

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 04 Mar 2011 22:33:35 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

DF> Tom Tromey <tromey@redhat.com> writes:
>> If somebody added the feature you want to Emacs, presumably it would
>> only be in Emacs 24, leaving you in the same situation.

DF> Point.  I would still prefer to depend on user-load-path or something,
DF> it strikes me as more general: you don't have to edit existing scripts.

Tom is talking about one thing (package management) and we're talking
about another (generic Lisp snippet management).  Sure, you can do the
latter with the former, but it's a long stretch and is fighting the
user.  Look, if I just want to put a file with

(setq myvar xyz)

or the el-get initialization, or whatever in the user-load-path, why do
I have to make a package out of it?

By analogy consider some of the software that lets you put a snippet in
a conf.d directory, obviously implying that this is convenient for the
user.  This is just a sampling from my machine:

apache2
newer crons (/etc/cron.d)
AppArmor
Grub
libpaper
sudo
logrotate
rsyslog
modprobe
sane
PAM

Why not provide the same level of convenience in Emacs?  Other than
security, what's the argument against it?  We all understand the
benefits of modularization made easy, right?

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]


On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
> 
> By analogy consider some of the software that lets you put a snippet in
> a conf.d directory, obviously implying that this is convenient for the
> user.  This is just a sampling from my machine:

Those are all machine-wide configurations.  For that, use site-lisp.

*Chad

[-- Attachment #2: Type: text/html, Size: 639 bytes --]

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

On Fri, Mar 04, 2011 at 09:18:22PM -0600, Ted Zlatanov wrote:
> On Fri, 04 Mar 2011 22:33:35 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 
> 
> DF> Tom Tromey <tromey@redhat.com> writes:
> >> If somebody added the feature you want to Emacs, presumably it would
> >> only be in Emacs 24, leaving you in the same situation.
> 
> DF> Point.  I would still prefer to depend on user-load-path or something,
> DF> it strikes me as more general: you don't have to edit existing scripts.
> 
> Tom is talking about one thing (package management) and we're talking
> about another (generic Lisp snippet management).  Sure, you can do the
> latter with the former, but it's a long stretch and is fighting the
> user.  Look, if I just want to put a file with
> 
> (setq myvar xyz)
> 
> or the el-get initialization, or whatever in the user-load-path, why do
> I have to make a package out of it?
> 
> By analogy consider some of the software that lets you put a snippet in
> a conf.d directory, obviously implying that this is convenient for the
> user.  This is just a sampling from my machine:
> 
> apache2
> newer crons (/etc/cron.d)
> AppArmor
> Grub
> libpaper
> sudo
> logrotate
> rsyslog
> modprobe
> sane
> PAM
> 
> Why not provide the same level of convenience in Emacs?  Other than
> security, what's the argument against it?  We all understand the
> benefits of modularization made easy, right?
> 
> Ted
> 
> 

There are ways to solve the problem you are looking at without wiring more
logic into Emacs. Use the .emacs file as a more sophisticated loader for a
complex configuration when necessary.

I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
I would look at these solutions first before proposing code that has to
be wired into the bootstrap codepath.

There are a number of issues a reliable loader should address:

how does it handle errors ?

how does it handle --batch ?

how does it handle --daemon ?

how simple,transparent,debuggable is the loader code ?

I have not seen any code so far in your proposal unless I accidentally
skipped or deleted a message with it.






[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Sat, 5 Mar 2011 11:11:34 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
>> 
>> By analogy consider some of the software that lets you put a snippet in
>> a conf.d directory, obviously implying that this is convenient for the
>> user.  This is just a sampling from my machine:

CB> Those are all machine-wide configurations.  For that, use site-lisp.

Bazaar has a plugins directory; files in it are automatically activated,
as an example of a user-level facility like this.  Anyhow, my point is
that placing a file in a directory is inherently more modular and
convenient to the user than augmenting a single file.  Do you disagree?

CB> If the user has to go through a set of steps to install and activate a
CB> package, how isn't that what ELPA does?

They are not packages, they are snippets of code.  ELPA requires far
more structure and many more steps.  For what I'm proposing, a lot less
work is required (just a y/n prompt the first time a snippet is found to
ensure it's not placed there maliciously).

On Sat, 5 Mar 2011 23:21:51 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> There are ways to solve the problem you are looking at without wiring more
MM> logic into Emacs. Use the .emacs file as a more sophisticated loader for a
MM> complex configuration when necessary.

I can and do, sure.  I don't think it creates a nice user experience.

MM> I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
MM> I would look at these solutions first before proposing code that has to
MM> be wired into the bootstrap codepath.

Thanks for pointing those out.  Would Grail or tidyconfig work as a core
option on startup (meaning the user doesn't have to install a package,
just customize a boolean)?  Do they handle signing the snippets the
first time they are found?  Can you give us a comparison of them and any
other similar solutions you know?  They have to be GPL and assigned to
Emacs to be included.

MM> There are a number of issues a reliable loader should address:

MM> how does it handle errors ?

MM> how does it handle --batch ?

MM> how does it handle --daemon ?

MM> how simple,transparent,debuggable is the loader code ?

MM> I have not seen any code so far in your proposal unless I accidentally
MM> skipped or deleted a message with it.

I can write the code but haven't seen the need for actual code yet (I
could have just comitted some code to Emacs instead, but find that
option to be antisocial).

I'd rather use something that already exists such as Grail or tidyconfig
that you kindly pointed out.  If you could tell us how any potential
solutions can help answer the questions above, that would be wonderful.
I can look at any solutions and evaluate them if you like.

To answer your questions from my perspective, IMHO a loader should work
like package.el.  IOW, whenever and however package.el is loaded
currently, this loader we're discussing should also be loaded, since
it's effectively a package manager but with snippets instead of
packages.

Ted

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 7, 2011, at 8:24 AM, Ted Zlatanov wrote:

> Bazaar has a plugins directory; files in it are automatically activated,
> as an example of a user-level facility like this.  Anyhow, my point is
> that placing a file in a directory is inherently more modular and
> convenient to the user than augmenting a single file.  Do you disagree?

My point is that the user must take an explicit step to do either, and
that placing a file in a magic directory and then answering questions
about it (both steps are required) is significantly less convenient
and equally or less modular than running an installer command.

So, yes, I do disagree.  What you call `augmenting a single file' can
be done with a simple, user-fiendly lisp command. The simple
combination of el-get and package.el is much of the way there today.

> CB> If the user has to go through a set of steps to install and activate a
> CB> package, how isn't that what ELPA does?
> 
> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).

> [...] from my perspective, IMHO a loader should work
> like package.el.  IOW, whenever and however package.el is loaded
> currently, this loader we're discussing should also be loaded, since
> it's effectively a package manager but with snippets instead of
> packages.

I think that you're conflating package.el with the (still developing)
policy concerning the management of the `official' central archive for
it once it's included by default. You say that you want something that
`should work like package.el', but I don't see where you say what your
problem with package.el is except to imply that you think some sort of
overhead will be too high.

I honestly doubt that there is a lot more effort required to create
something that works with package.el than there is to make a snippet
that works automatically with just a simple mechanical checksum
test. Put another way: I doubt that the overhead is likely to be too
high for anyone who's actually creating something that wants to be
shared automatically.  Put another another way, I think that the
snippet of code that can't be a package or in site-lisp but that
several users want to share and use automatically is really unlikely
to exist.

Maybe you could take a look at package.el start with
http://tromey.com/elpa/faq.html and describe what you think is too
much effort? Package.el will be included with emacs, can be (is
being?) developed to make it more user-friendly, and doesn't have the
security issues of the magic directory.

*Chad

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 7 Mar 2011 09:18:44 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> On Mar 7, 2011, at 8:24 AM, Ted Zlatanov wrote:
>> Bazaar has a plugins directory; files in it are automatically activated,
>> as an example of a user-level facility like this.  Anyhow, my point is
>> that placing a file in a directory is inherently more modular and
>> convenient to the user than augmenting a single file.  Do you disagree?

CB> My point is that the user must take an explicit step to do either, and
CB> that placing a file in a magic directory and then answering questions
CB> about it (both steps are required) is significantly less convenient
CB> and equally or less modular than running an installer command.

CB> So, yes, I do disagree.  What you call `augmenting a single file' can
CB> be done with a simple, user-fiendly lisp command. The simple
CB> combination of el-get and package.el is much of the way there today.

Again you're talking about package installation.  I want to create
$LOAD_DIR/000-keys.el and have it loaded on startup.  I don't want it
installed, managed, or wrapped.  This mechanism will facilitate
bootstrapping el-get, for instance, but it will also enable users to
manage their configurations better.

CB> I think that you're conflating package.el with the (still developing)
CB> policy concerning the management of the `official' central archive for
CB> it once it's included by default. You say that you want something that
CB> `should work like package.el', but I don't see where you say what your
CB> problem with package.el is except to imply that you think some sort of
CB> overhead will be too high.

I was talking about the mechanics as far as Matt's questions: how it
behaves on --batch, etc.  I was not suggesting it should do what
package.el does in terms of functionality.

I don't have a problem with package.el.  I just don't want to have to
create a package in order to have a modular snippet as I described
above.  Why can't the two coexist?

CB> I honestly doubt that there is a lot more effort required to create
CB> something that works with package.el than there is to make a snippet
CB> that works automatically with just a simple mechanical checksum
CB> test. Put another way: I doubt that the overhead is likely to be too
CB> high for anyone who's actually creating something that wants to be
CB> shared automatically.  Put another another way, I think that the
CB> snippet of code that can't be a package or in site-lisp but that
CB> several users want to share and use automatically is really unlikely
CB> to exist.

Who said anything about sharing?  I want to break down my .emacs for
example; why can't I do it by putting 8 files in a directory instead of
writing (load...) 8 times in my .emacs?  Here's that portion of my
current .emacs:

(load "tzz.emacs.start.el")
(load "tzz.emacs.functions.el")
(load "tzz.emacs.libraries.el")
(load "tzz.emacs.colors.el")
(load "tzz.emacs.options.el")
(load "tzz.emacs.keys.el")
(load "tzz.emacs.custom.el")
(load "tzz.emacs.skeletons.el")

I want, instead, to customize load-dir to ~/emacs.d/conf.d and put those
8 files in it.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chad Brown <yandros@MIT.EDU> writes:
> Put another another way, I think that the
> snippet of code that can't be a package or in site-lisp but that
> several users want to share and use automatically is really unlikely
> to exist.

Ever heard about emacswiki?

What do you think the chances are that authors of snippets and scripts
shared over at emacswiki get to package them to ELPA?  My take is very
very very low, low enough that I wrote el-get just so that I don't have
to take the chance.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 8, 2011, at 9:36 AM, Dimitri Fontaine wrote:

> Chad Brown <yandros@MIT.EDU> writes:
>> Put another another way, I think that the
>> snippet of code that can't be a package or in site-lisp but that
>> several users want to share and use automatically is really unlikely
>> to exist.
> 
> Ever heard about emacswiki?

I think that you haven't been following the conversation closely enough to understand what ``share and use automatically'' means.  The `automatically' part of that sentence is important.

It turns out that what Ted really wants is a way to load all the elisp files in a directory without listing them explicitly. The entire security, package, and downloader discussion (in which I suggested using el-get) was a side-effect of premature overgeneralization.

I hope that helps,
*Chad

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).

There's no reason why package.el can't accept any random elisp file.


        Stefan

Re: user-controlled load-path extension: load-dir

[PJ Weisberg]

On 3/7/11, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> They are not packages, they are snippets of code.  ELPA requires far
>> more structure and many more steps.  For what I'm proposing, a lot less
>> work is required (just a y/n prompt the first time a snippet is found to
>> ensure it's not placed there maliciously).
>
> There's no reason why package.el can't accept any random elisp file.

I think this proposal is really about code snippets that people would
otherwise just cut and paste into their .emacs file.  The average
user's .emacs often winds up containing mostly code they found
somewhere and use without really understanding it.  Dropping each
snippit in its own file would be a big help if the user ever did need
to debug some problem with his init, or if he decided one day to
actually learn elisp.

(I din't know if the security/hash thing is really necessary.  If the
attacker already has write access to your home directory, how much
worse can it get?)

-PJ

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> I think this proposal is really about code snippets that people would
> otherwise just cut and paste into their .emacs file.  The average
> user's .emacs often winds up containing mostly code they found
> somewhere and use without really understanding it.  Dropping each
> snippit in its own file would be a big help if the user ever did need
> to debug some problem with his init, or if he decided one day to
> actually learn elisp.

I'm far from convinced it's better for people to drop random chunks of
configuration into separate files rather than all in the .emacs file.

If you're talking about downloading random files containing
configuration code (so the user doesn't have to create a file, inventing
a file name for it), then maybe that could make sense, but I haven't
seen such files very much, and I suspect many of them could easily be
turned either into themes or into packages installable via package.el.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 07 Mar 2011 21:46:10 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I think this proposal is really about code snippets that people would
>> otherwise just cut and paste into their .emacs file.  The average
>> user's .emacs often winds up containing mostly code they found
>> somewhere and use without really understanding it.  Dropping each
>> snippit in its own file would be a big help if the user ever did need
>> to debug some problem with his init, or if he decided one day to
>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

It's not "better," it's what people already do.  Emacs would just make
it easier than the current situation, which is to drop a file plus edit
the .emacs every time a file is added or deleted.  What other choice is
there to modularize .emacs?

SM> If you're talking about downloading random files containing
SM> configuration code (so the user doesn't have to create a file, inventing
SM> a file name for it), then maybe that could make sense, but I haven't
SM> seen such files very much, and I suspect many of them could easily be
SM> turned either into themes or into packages installable via package.el.

Obviously that works for package managers like el-get.

OK, let's say I have the 8 files I listed earlier (tzz.emacs.*.el).  I
want to load them modularly.  So I put them in the load-dir.  Do I have
to make 8 packages?  And every time I update one of those files, do I
have to repackage it with a new version?  That seems workable but
tedious compared to the code proposed by Ben Key and Evans Winner.

If you're against including the load-dir in the core and enabling it by
default, how about making it a GNU ELPA package so it's easily
installable?

On Tue, 8 Mar 2011 01:14:37 -0600 Ben Key <bkey76@gmail.com> wrote: 

BK> I am attaching to this message a quick implementation of this feature I
BK> threw together in about an hour.  I am not familiar with the Emacs package
BK> mechanism but I am certain that this could be converted to an Emacs package
BK> without too much work thus making it easier to install.

On Tue, 08 Mar 2011 02:58:03 -0700 Evans Winner <ego111@gmail.com> wrote: 

EW> For what it's worth, this is the thing I have in my .emacs.
EW> It does what I want it to, which is just to allow me to
EW> break my .emacs into various not-excessively-long files
EW> organized by subject, like email config, org-mode config and
EW> so-forth.

Thanks for your code.  Both of your solutions are similar to what I
would like to use, but Mike Mattie's concerns about startup behavior
(mainly for daemon mode, for --batch mode, and when there are errors)
are valid.  The recursion behavior should probably not be the default by
parallel with other such systems I listed, e.g. /etc/rc.d and Bazaar's
plugins directory.  It's easy enough to make it optional or to write a
snippet that loads a specific subdirectory.  The argument I have against
recursion is that it's hard to tell at a glance what's going to get
loaded.

It seems no one thinks prompting about new or changed files in the
load-dir is worthwhile, so never mind about that part of my proposal.

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?

It's a bit too trivial to be a package.

Just to be specific: are you simply asking for a function to iterates
through the files in a directory loading them?  Or is it more
complicated than that?

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 08 Mar 2011 12:14:52 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you're against including the load-dir in the core and enabling it by
>> default, how about making it a GNU ELPA package so it's easily
>> installable?

CY> It's a bit too trivial to be a package.

CY> Just to be specific: are you simply asking for a function to iterates
CY> through the files in a directory loading them?  Or is it more
CY> complicated than that?

I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
when set to a list of directories, will tell Emacs to load all the files
in the directory without recursion.  The default should be nil.  This
should not require .emacs modifications.

I suggested packaging it as an alternative in case it was not allowed in
the Emacs core.  I realize it seems trivial but it's really convenient
for managing snippets, ELisp files, and modularizing .emacs.

On Tue, 8 Mar 2011 10:30:14 -0800 Chad Brown <yandros@MIT.EDU> wrote: 

CB> It turns out that what Ted really wants is a way to load all the elisp
CB> files in a directory without listing them explicitly. The entire
CB> security, package, and downloader discussion (in which I suggested
CB> using el-get) was a side-effect of premature overgeneralization.

If this makes it into Emacs, the effect for Dimitri's el-get will be to
simplify the installer and make it Just Work, without modifying the
load-path.  In fact installing any ELisp .el file *manually* will be
simpler.  Helping el-get was the reason I started this thread, but I
have selfish reasons too: it would make my life much easier.

el-get may choose to use this facility directly as well (to manage
EmacsWiki recipes, for instance), but I don't know Dimitri's plans and
it already manages recipes in subdirectories.

Ted

Re: user-controlled load-path extension: load-dir

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Tue, Mar 08 2011, Ted Zlatanov wrote:

> I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
> when set to a list of directories, will tell Emacs to load all the files
> in the directory without recursion.  The default should be nil.  This
> should not require .emacs modifications.

Hum, I'd like to have that defcustom (or another one) to have a
directory where Emacs would not stupidly load the file, but build an
autoload file for them and load that one.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>> I'm asking for a defcustom, e.g. `load-dir', *in the Emacs core* which,
>> when set to a list of directories, will tell Emacs to load all the files
>> in the directory without recursion.  The default should be nil.  This
>> should not require .emacs modifications.

> Hum, I'd like to have that defcustom (or another one) to have a
> directory where Emacs would not stupidly load the file, but build an
> autoload file for them and load that one.

That would make more sense, indeed.  That's also closer to what
package.el (w|sh)ould do with random .el files.


        Stefan

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Hum, I'd like to have that defcustom (or another one) to have a
>> directory where Emacs would not stupidly load the file, but build an
>> autoload file for them and load that one.
>
> That would make more sense, indeed.  That's also closer to what
> package.el (w|sh)ould do with random .el files.

If said random .el file follows the Emacs Lisp file conventions, and has
a `;; Version: XX.XX.XX' line in its header section, it's already a
valid package.  M-x package-install-file will do the right thing.

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>>> I think this proposal is really about code snippets that people would
>>> otherwise just cut and paste into their .emacs file.  The average
>>> user's .emacs often winds up containing mostly code they found
>>> somewhere and use without really understanding it.  Dropping each
>>> snippit in its own file would be a big help if the user ever did need
>>> to debug some problem with his init, or if he decided one day to
>>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

> It's not "better," it's what people already do.

I must be missing something.  I see two different things on the web:
packages on the one hand and Elisp configuration chunks on the other.
The first come in files you can download, the other comes in
<pre>...</pre> elements in HTML pages which you can copy&paste.
My claim is that the first can be better handled by package.el or themes
and the other works just fine in .emacs.

> Emacs would just make it easier than the current situation, which is
> to drop a file plus edit the .emacs every time a file is added or
> deleted.

If you drop the file with package.el you don't need to edit the
.emacs correspondingly.

> want to load them modularly.  So I put them in the load-dir.  Do I have
> to make 8 packages?  And every time I update one of those files, do I
> have to repackage it with a new version?  That seems workable but
> tedious compared to the code proposed by Ben Key and Evans Winner.

If you're really talking about configuration code rather than packages,
then I tend to assume that people whose .emacs is so large as to need
modularization can figure out which kind of modularization they want and
implement (or copy&paste) the corresponding loop to load the various files.

I'm not even sure why you'd want to "modularize" in this way: my .emacs
was fairly large but splitting it into separate files never seemed like
a good way to help, since I'd then have to figure out how to make C-s
and M-/ find matches in neighboring files.  Instead I "split" it with
outline-minor-mode.

> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?

Such a package would be perfectly acceptable, yes.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> I think this proposal is really about code snippets that people would
>>>> otherwise just cut and paste into their .emacs file.  The average
>>>> user's .emacs often winds up containing mostly code they found
>>>> somewhere and use without really understanding it.  Dropping each
>>>> snippit in its own file would be a big help if the user ever did need
>>>> to debug some problem with his init, or if he decided one day to
>>>> actually learn elisp.

SM> I'm far from convinced it's better for people to drop random chunks of
SM> configuration into separate files rather than all in the .emacs file.

>> It's not "better," it's what people already do.

SM> I must be missing something.  I see two different things on the web:
SM> packages on the one hand and Elisp configuration chunks on the other.
SM> The first come in files you can download, the other comes in
SM> <pre>...</pre> elements in HTML pages which you can copy&paste.
SM> My claim is that the first can be better handled by package.el or themes
SM> and the other works just fine in .emacs.

There is a third class: streams (or snippets) of code I don't want to
keep inside my .emacs because I don't like to make it big.  There are
some users who will use and enjoy this functionality, even if other
users won't.  As I proposed it, it will be off by default so it only
benefits those who enable it and won't harm those who don't.

This kind of modularization doesn't make it to the web as much because
it's usually specific to the user's needs.  Here are some examples:

http://www.emacswiki.org/emacs/JoelAdamson#toc5
http://www.io.com/~jimm/emacs_tips.html#my-dot-emacs
http://www.emacswiki.org/emacs/DotEmacsModular
https://github.com/technomancy/emacs-starter-kit
http://www.emacswiki.org/emacs/InitSplit
http://stackoverflow.com/questions/2079095/how-to-modularize-an-emacs-configuration

It's a common need as you can see.

In the Emacs core it will be easier to enable it, that's all.  Otherwise
the instructions have to be "install package load-dir, THEN do the rest"
which is a bit more work, requires a network connection, etc.

>> want to load them modularly.  So I put them in the load-dir.  Do I have
>> to make 8 packages?  And every time I update one of those files, do I
>> have to repackage it with a new version?  That seems workable but
>> tedious compared to the code proposed by Ben Key and Evans Winner.

SM> If you're really talking about configuration code rather than packages,
SM> then I tend to assume that people whose .emacs is so large as to need
SM> modularization can figure out which kind of modularization they want and
SM> implement (or copy&paste) the corresponding loop to load the various files.

Yes, but you're assuming managing configuration modules in monolithic
Emacs Lisp is the best way.  Give us something simple and easy to manage
the loop at the filesystem level, so we don't have to write it
ourselves.

SM> I'm not even sure why you'd want to "modularize" in this way: my .emacs
SM> was fairly large but splitting it into separate files never seemed like
SM> a good way to help, since I'd then have to figure out how to make C-s
SM> and M-/ find matches in neighboring files.  Instead I "split" it with
SM> outline-minor-mode.

For me it works better.  I like small files; outline-minor-mode and
folding-mode don't work for me.  I suspect I'm not the only one.  See
the URLs above for a list of similar needs.

>> If you're against including the load-dir in the core and enabling it by
>> default, how about making it a GNU ELPA package so it's easily
>> installable?

SM> Such a package would be perfectly acceptable, yes.

I'll work with Ben Key to do this in a package if you and Chong Yidong
reject this functionality from the Emacs core.

Ted

Re: user-controlled load-path extension: load-dir

[Jan D.]

Ted Zlatanov skrev 2011-03-08 22:38:
> On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier<monnier@IRO.UMontreal.CA>  wrote:
>
> SM>  If you're really talking about configuration code rather than packages,
> SM>  then I tend to assume that people whose .emacs is so large as to need
> SM>  modularization can figure out which kind of modularization they want and
> SM>  implement (or copy&paste) the corresponding loop to load the various files.
>
> Yes, but you're assuming managing configuration modules in monolithic
> Emacs Lisp is the best way.  Give us something simple and easy to manage
> the loop at the filesystem level, so we don't have to write it
> ourselves.
>
> SM>  I'm not even sure why you'd want to "modularize" in this way: my .emacs
> SM>  was fairly large but splitting it into separate files never seemed like
> SM>  a good way to help, since I'd then have to figure out how to make C-s
> SM>  and M-/ find matches in neighboring files.  Instead I "split" it with
> SM>  outline-minor-mode.
>
> For me it works better.  I like small files; outline-minor-mode and
> folding-mode don't work for me.  I suspect I'm not the only one.  See
> the URLs above for a list of similar needs.
>

As others have said, I was also suprised that dropping a .el-fil in 
~/emacs.d didn't load it by default.  My emacs is quite big and I'd love 
to split it up.  Another thing I really would like is for customize to 
save its stuff in its own file and not in my .emacs.  With a load-dir 
this would be trivial.

There is a real need an packages isn't it.  Its like wanting a rowing 
boat and told to use the latest battle ship instead.

	Jan D.

Re: user-controlled load-path extension: load-dir

[Christoph Scholtes]

On 3/9/2011 12:06 AM, Jan D. wrote:

> Another thing I really would like is for customize to
> save its stuff in its own file and not in my .emacs. With a load-dir
> this would be trivial.

How about this:

(setq custom-file "~/.emacs.d/init-custom.el")
(load custom-file)

See documentation (C-h v custom-file) for details.

Christoph

Re: user-controlled load-path extension: load-dir

[Jan Djärv]

Christoph Scholtes skrev 2011-03-09 08.17:
> On 3/9/2011 12:06 AM, Jan D. wrote:
>
>> Another thing I really would like is for customize to
>> save its stuff in its own file and not in my .emacs. With a load-dir
>> this would be trivial.
>
> How about this:
>
> (setq custom-file "~/.emacs.d/init-custom.el")
> (load custom-file)
>

Yes, but this is missing the point.  I can make an load-dir feature in .emacs 
too.  It is having this in emacs core, working without extra user work that is 
the point.

	Jan D.

`custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Drew Adams]

> >> Another thing I really would like is for customize to
> >> save its stuff in its own file and not in my .emacs. With 
> >> a load-dir this would be trivial.
> >
> > How about this:
> > (setq custom-file "~/.emacs.d/init-custom.el")
> > (load custom-file)
> 
> It is having this in emacs core, working without extra 
> user work that is the point.

My impression is that many Emacs (most?) users are unaware of `custom-file'.
That's too bad, IMO.

I think Emacs should use a separate custom file _by default_ (empty initially),
and thus, by default, keep separate (a) the direct user editing and (b) the
automatic modifications done by Customize.

I would be in favor of having `custom-file' default to ~/.emacs-custom.el or
~/.emacs.d/user-custom.el or something.

By itself, that wouldn't clue users in.  A separate question would be whether to
also have an (almost) empty `.emacs' by default, containing just an explicit
load of `custom-file'.  I'd say yes.

Even if we did not provide a (nearly empty) init file and an (empty)
`custom-file' by default, documenting the use of `custom-file' at the same place
where we document `.emacs' (node `Init File') would be helpful, IMO.  Even just
adding a cross reference there plus a brief mention of `custom-file' would be
helpful.  Today, the `custom-file' doc is buried in the Customization section,
with no mention in the init-file section.

If users did have a default init file and `custom-file' (the former loading the
latter, the latter being empty), then they would more easily keep separate their
own Lisp customizations from any customizations made by Customize.  Mixing the
two is error prone (hence the warning comment we insert today).

Users would be free to add their own Lisp customizations before or after the
load of `custom-file'.  They would also be free to define `custom-file'
conditionally in the init file.  And they could remove the load of `custom-file'
altogether or move it to some other file that gets loaded (i.e. move the load
away from the top level).

IOW, they could do everything they do today.  The only changes would be that by
_default_:

1. Users would have an init file (~/.emacs).

2. The init file would just load `custom-file' (explicitly).  Otherwise it would
be empty.

3. `custom-file' would default to ~/.emacs-custom.el or
~/.emacs.d/user-custom.el or similar (TBD).

Except for #3, I've suggested something similar before (thread "always put
Customizations in `custom-file', never in `user-init-file',
http://lists.gnu.org/archive/html/emacs-devel/2007-12/msg00567.html).

At that time, some were in favor, a few were opposed.  Richard turned it down in
favor of fixing bugs for Emacs 22.  XEmacs apparently already has something like
this (see the thread).

A variant of what's proposed above would be to not have a default init file with
an explicit load of `custom-file', but to just load `custom-file' automatically
after the init file if it is not loaded before then.  This was discussed in the
thread as well.  That too would be OK by me.  (See the thread for details.)

The discussion in that thread also dealt with the possibility of automigrating
existing Customize customizations from .emacs to `custom-file'.  Leaving out
migration altogether - or at least making it optional (on-demand) - would
perhaps lead us to an easier consensus about using `custom-file' by default. 

Perhaps this topic is worth revisiting?

`custom-file' and init-file

[Stephen J. Turnbull]

Drew Adams writes:

 > I would be in favor of having `custom-file' default to
 > ~/.emacs-custom.el or ~/.emacs.d/user-custom.el or something.

FWIW, XEmacs uses the following scheme.

(1) The startup code automatically loads custom-file, if non-nil and
    it exists, *after* loading the init file, if that exists.
    (custom-file will be loaded if init file doesn't exist.)
(2) custom-file defaults to (the equivalent of) .emacs.d/custom.el.

Note that the custom file does not need to be on the load-path since
it has its own location variable, so the name conflict doesn't cause a
problem.  I've not heard of a real issue about the name in 15 years of
use of this scheme.  My own experience is that it's only an issue if
I'm working on lisp/custom.el, in which case it's often prompted by
some problem with ~/.xemacs/custom.el, and then the buffer name
conflict is mildly annoying.

 > By itself, that wouldn't clue users in.  A separate question would
 > be whether to also have an (almost) empty `.emacs' by default,
 > containing just an explicit load of `custom-file'.  I'd say yes.

Note that having a explicit load would require either a conditional or
that custom-file exist.  The XEmacs scheme makes this set of issues
moot.

The only real problem with the XEmacs scheme is that some people have
init files that depend on the custom file.  In that case the remedy is
somewhat ugly but effective.  Add

(load custom-file)
(add-hook 'after-init-hook `(lambda () (setq custom-file ,custom-file)))
(setq custom-file nil)

early enough.

Re: `custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Evans Winner]

,------ Drew Adams wrote ------

|    By itself, that wouldn't clue users in.  A separate
|    question would be whether to also have an (almost)
|    empty `.emacs' by default, containing just an explicit
|    load of `custom-file'.  I'd say yes.

Out of curiosity, would this complicate the process of Emacs
installation?  Is the idea that the first time a user runs
Emacs, it checks to see if there is a .emacs in (getenv
"HOME"), and if not, creates it?  Would that complicate
things for system administrators?  Presumably they should
use site-init.el, but I know some add some kind of simple
.emacs file to the user account creation routine.

By the way, for the default location of the custom file, I
would suggest ~/.emacs.d.  Right now my ~/.emacs.d is filled
with things put there by elpa, predictive, org-mode, rcirc,
my bookmarks file is there.  I used to put my own one-off
third-party libraries there, but I gave up and now put them
in ~/emacs.  I also put my own hacks in ~/.emacs-local and
use the `load-dir' function I posted to load all that.
Anyway, the convention seems to be that things
auto-generated by Emacs and such things belong in
~/.emacs.d.

|    Users would be free to add their own Lisp
|    customizations before or after the load of
|    `custom-file'.  

I think this is important.  I only use the customize
interface, really, when I don't understand something well
enough to handle it in my .emacs.  Actually, there's not
much left in my custom file now, because I tend to move
things out of it to my own init routines as soon as I do
understand something.  I like to feel like I know what is
going on, and like I can change things just by typing in a
buffer, instead of fiddling with the customize interface.
The result, and I can't be alone in this, is that I make the
call to load the custom file at the beginning of my .emacs,
so that if I happen to have defined something twice by
mistake (or because I am too lazy to fix it) the result will
be whatever I have defined in my .emacs.

RE: `custom-file' and init-file [was: user-controlled load-pathextension: load-dir]

[Drew Adams]

> |    By itself, that wouldn't clue users in.  A separate
> |    question would be whether to also have an (almost)
> |    empty `.emacs' by default, containing just an explicit
> |    load of `custom-file'.  I'd say yes.
> 
> Out of curiosity, would this complicate the process of Emacs
> installation?

Dunno.  Don't think so.

> Is the idea that the first time a user runs
> Emacs, it checks to see if there is a .emacs in (getenv
> "HOME"), and if not, creates it?

Yes.  That is one possibility.
(Of course, emacs -Q would act as now: ignore any init file.)

> Would that complicate things for system administrators?

Dunno.  Why would it?

> Presumably they should
> use site-init.el, but I know some add some kind of simple
> .emacs file to the user account creation routine.

Well if the sysadmins give their users a standard (for their organization)
default .emacs, then that would be used, no?  IOW, presumably they or the user
would put that file in the user's $HOME, so it would be picked up by Emacs -
same as now.  I don't see anything changing here.

> By the way, for the default location of the custom file, I
> would suggest ~/.emacs.d.

Sounds good to me.

But see also the other possibility discussed in the referenced thread and
mentioned again today by Stephen: we autoload `custom-file' after the .emacs,
etc.  Either way is fine with me, and there are no doubt other possibilities.

The basic idea is to give the user a `custom-file' by default (empty), and have
Emacs automatically load it.  Users would still have control over whether and
when to load it.

Customizations by Customize would always go to the `custom-file', unless
`custom-file' raises an error.  (Or is some particular value - see the thread
for other possibilities here.)  The point is that we would make sure that users
would still have the possibility of not using `custom-file' at all and thus
having Customize write to .emacs instead. That just would not be the default
behavior.

Re: `custom-file' and init-file [was: user-controlled load-path extension: load-dir]

[Stephen J. Turnbull]

Evans Winner writes:

 > Out of curiosity, would this complicate the process of Emacs
 > installation?

That depends on whether the recommendation to use a separate
custom-file is strong enough to put in an automatic "migration"
process.  XEmacs did that; many users went to the trouble of
acknowledging its convenience, but other users suffered data loss
while converting .emacs to XEmacs conventions.  That said, such data
loss is rather unlikely for Emacs users (the problem occurs if the
init file signals while loading; this is unlikely in Emacs -- few
users mess with installing a new Emacs while their .emacs is
non-working! -- but is somewhat likely if you load an .emacs designed
for Emacs into XEmacs).

 > Is the idea that the first time a user runs
 > Emacs, it checks to see if there is a .emacs in (getenv
 > "HOME"), and if not, creates it?

In Drew's scheme, that would probably be the way to go.  (This is very
safe compared to "migrating" an existing .emacs.)  In the XEmacs
scheme (which normally loads custom-file after the init file), it's
unnecessary.

 > Would that complicate things for system administrators?

Slightly.  Probably not a big deal though.  The .emacs file is
explicitly for users to customize, so system administrators should not
count on it being present or on it containing particular code.  That's
what the site init and default files are for.

 > The result, and I can't be alone in this, is that I make the
 > call to load the custom file at the beginning of my .emacs,
 > so that if I happen to have defined something twice by
 > mistake (or because I am too lazy to fix it) the result will
 > be whatever I have defined in my .emacs.

This requires a bit of extra one-time effort in the XEmacs scheme
(which implicitly loads custom-file after the init file).  We consider
it worth it because of the convenience and transparency of the process
for almost all users.

Since the snippet used is entirely generic, it could be wrapped up in
a function `load-custom-file-now' or similar.

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Jan Djärv writes:

 > Christoph Scholtes skrev 2011-03-09 08.17:
 > > On 3/9/2011 12:06 AM, Jan D. wrote:
 > >
 > >> Another thing I really would like is for customize to
 > >> save its stuff in its own file and not in my .emacs. With a load-dir
 > >> this would be trivial.

 > > How about this:

 > > (setq custom-file "~/.emacs.d/init-custom.el")
 > > (load custom-file)

 > Yes, but this is missing the point.  I can make an load-dir feature
 > in .emacs too.  It is having this in emacs core, working without
 > extra user work that is the point.

Yeah, except that as proposed you'll have to enable it in the init
file anyway since it's off by default.

It's also not entirely trivial, as experience in XEmacs shows that the
decision about whether to load such things (specifically custom-file)
*before* or *after* the user init file is a delicate one.  I can only
imagine that a load-dir would very likely be the source of numerous
bugs as some snippets conflict with or depend on others but the
automatic loader gets them in the wrong order.

Note also that .d directories generally use some convention (such as
file names starting with fixed width integers) that ensure that
snippets sort in the right order.  More modern systems use (please sit
down, you're in for a shock) full-blown dependency systems (requires,
provides, conflicts, etc) in the snippets, which avoids the need to
maintain explicit order in favor of a partial order.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> I can only imagine that a load-dir would very likely be the source
SJT> of numerous bugs as some snippets conflict with or depend on others
SJT> but the automatic loader gets them in the wrong order.

We're just adding a mechanism for easy personal code deployment and
modularization, not a cure for broken code.  If there's a load order
conflict the user will have to figure it out, just like putting loads in
the wrong order in the .emacs file.

SJT> Note also that .d directories generally use some convention (such as
SJT> file names starting with fixed width integers) that ensure that
SJT> snippets sort in the right order.

I'm sure users can use that (I will).  But they can also specify the
order with explicit loads if that's what they need.

SJT> More modern systems use (please sit down, you're in for a shock)
SJT> full-blown dependency systems (requires, provides, conflicts, etc)
SJT> in the snippets, which avoids the need to maintain explicit order
SJT> in favor of a partial order.

That's, again, overengineering the problem (as Mark Twain put it, "with
all the modern inconveniences").

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
 > 
 > SJT> I can only imagine that a load-dir would very likely be the source
 > SJT> of numerous bugs as some snippets conflict with or depend on others
 > SJT> but the automatic loader gets them in the wrong order.
 > 
 > We're just adding a mechanism for easy personal code deployment and
 > modularization, not a cure for broken code.

There's no broken code here, just code not engineered with automatic
loading in mind.  Ie, it's the idea of automatically loading random
snippets that is broken: it's a mechanism for speeding up the mess, as
Robert Townsend put it.  I don't think that belongs in core, but
rather should be an individually maintained function, tuned to each
person's requirements.

As for easy personal code deployment and modularization, there's
nothing that says that deployment and modularization requires
implementation as separate files, and in fact using separate files for
most things that you would do in an init file is an annoyance as
Stefan points out.  Stefan's recommendation of using outline minor
mode easily scales to a 100kB file for me in other contexts, I imagine
it would work fine for the init file.  While I admit YMMV as a user,
from the point of view of Emacs maintainers this blows away the
load-dir idea in terms of maintainability of Emacs core.

Note that the good reasons for using separate files (assuming you use
an editor that supports things like outline minor mode -- you do,
don't you?<wink>) are not really compatible with automatic
installation and loading.  They're things like lazy loading and
optional loading to avoid memory bloat and namespace pollution.  Lazy
loading could be a win (somebody mentioned autoloading instead of
automatic loading at init time), therefore, but guess what? that
requires reading code and deciding where to put autoload cookies.

 > SJT> Note also that .d directories generally use some convention (such as
 > SJT> file names starting with fixed width integers) that ensure that
 > SJT> snippets sort in the right order.
 > 
 > I'm sure users can use that (I will).  But they can also specify the
 > order with explicit loads if that's what they need.

Neither of those is easy or automatic.  To the extent that such are
needed, people are going to have to sit down and read code to figure
out what the problems are.  That's precisely what the system is
supposed to avoid.

Note that you also have a problem that such explicitly loaded files
will often need to be moved out of the load-dir, as they won't be
designed to be idempotent.  Mostly not a problem, of course, and
therefore all the more annoying and confusing when it does arise.
 
 > SJT> More modern systems use (please sit down, you're in for a shock)
 > SJT> full-blown dependency systems (requires, provides, conflicts, etc)
 > SJT> in the snippets, which avoids the need to maintain explicit order
 > SJT> in favor of a partial order.
 > 
 > That's, again, overengineering the problem (as Mark Twain put it, "with
 > all the modern inconveniences").

Well, no, it's not.  The problem in question is maintaining a .d
directory without requiring the user to have a clue.  If Stefan and
Yidong could get away with "Ted Z says that's overengineering so we
ain't gonna touch it", it might be worth putting this in core.  But my
recommendation to them is "snippet conflicts are going to be a
persistent annoyance in the long term; let Ted Z and Jan D maintain a
package and deal with the PEBKACs and RFEs."

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 04:33:16 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> On Thu, 10 Mar 2011 02:29:02 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
>> 
SJT> I can only imagine that a load-dir would very likely be the source
SJT> of numerous bugs as some snippets conflict with or depend on others
SJT> but the automatic loader gets them in the wrong order.
>> 
>> We're just adding a mechanism for easy personal code deployment and
>> modularization, not a cure for broken code.

SJT> There's no broken code here, just code not engineered with automatic
SJT> loading in mind.

By your logic out-of-order startup scripts in /etc/rc.d should be
managed with an external system because simple shell scripts are not
engineered with automatic loading in mind.  Sometimes things really are
independent and you have to trust the user.

SJT> Ie, it's the idea of automatically loading random snippets that is
SJT> broken: it's a mechanism for speeding up the mess, as Robert
SJT> Townsend put it.  I don't think that belongs in core, but rather
SJT> should be an individually maintained function, tuned to each
SJT> person's requirements.

OK, thank you for your opinion.

SJT> As for easy personal code deployment and modularization, there's
SJT> nothing that says that deployment and modularization requires
SJT> implementation as separate files, and in fact using separate files for
SJT> most things that you would do in an init file is an annoyance as
SJT> Stefan points out.  Stefan's recommendation of using outline minor
SJT> mode easily scales to a 100kB file for me in other contexts, I imagine
SJT> it would work fine for the init file.  While I admit YMMV as a user,
SJT> from the point of view of Emacs maintainers this blows away the
SJT> load-dir idea in terms of maintainability of Emacs core.

OK.  What reasons do you have to NOT include the load-dir functionality
in the core if it's disabled by default and must be consciously enabled?
That you're happy without it?

SJT> Note that the good reasons for using separate files (assuming you use
SJT> an editor that supports things like outline minor mode -- you do,
SJT> don't you?<wink>) are not really compatible with automatic
SJT> installation and loading.  They're things like lazy loading and
SJT> optional loading to avoid memory bloat and namespace pollution.  Lazy
SJT> loading could be a win (somebody mentioned autoloading instead of
SJT> automatic loading at init time), therefore, but guess what? that
SJT> requires reading code and deciding where to put autoload cookies.

For me those reasons are modularization and easy code deployment on a
personal level.  I think I've been clear on that and wonder how you
determined your reasons were the good ones.

SJT> Note also that .d directories generally use some convention (such as
SJT> file names starting with fixed width integers) that ensure that
SJT> snippets sort in the right order.
>> 
>> I'm sure users can use that (I will).  But they can also specify the
>> order with explicit loads if that's what they need.

SJT> Neither of those is easy or automatic.  To the extent that such are
SJT> needed, people are going to have to sit down and read code to figure
SJT> out what the problems are.  That's precisely what the system is
SJT> supposed to avoid.

Who said load-dir is supposed to avoid problems?  As with the earlier
comment about broken code, if you screw up your configuration layout in
any form, you go and fix it.  Emacs should indicate where the error
happened, but beyond that it's a user issue.

SJT> Note that you also have a problem that such explicitly loaded files
SJT> will often need to be moved out of the load-dir, as they won't be
SJT> designed to be idempotent.  Mostly not a problem, of course, and
SJT> therefore all the more annoying and confusing when it does arise.

Sorry, I don't follow.  Can you give a specific example?
 
SJT> More modern systems use (please sit down, you're in for a shock)
SJT> full-blown dependency systems (requires, provides, conflicts, etc)
SJT> in the snippets, which avoids the need to maintain explicit order
SJT> in favor of a partial order.
>> 
>> That's, again, overengineering the problem (as Mark Twain put it, "with
>> all the modern inconveniences").

SJT> Well, no, it's not.  The problem in question is maintaining a .d
SJT> directory without requiring the user to have a clue.

NO IT'S NOT!  Why are you assuming you must treat the user like an idiot
who doesn't know he put a file in a directory and can't figure out load
dependencies?  If the user wants hand-holding, he will use Customize and
package.el and el-get and all the other marvels of the modern age.  I
prefer not to, for my own snippets I will write.

SJT> If Stefan and Yidong could get away with "Ted Z says that's
SJT> overengineering so we ain't gonna touch it", it might be worth
SJT> putting this in core.  But my recommendation to them is "snippet
SJT> conflicts are going to be a persistent annoyance in the long term;
SJT> let Ted Z and Jan D maintain a package and deal with the PEBKACs
SJT> and RFEs."

OK, got it.  Thanks.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SJT> There's no broken code here, just code not engineered with automatic
 > SJT> loading in mind.
 > 
 > By your logic out-of-order startup scripts in /etc/rc.d should be
 > managed with an external system because simple shell scripts are not
 > engineered with automatic loading in mind.

Yes.  That's what all of the GNU/Linux distros actually do.  IIRC,
NetBSD does as well, as does Mac OS X (nee FreeBSD), and *BSD is quite
a bit farther to the "user responsibility" side of the field than
GNU/Linux distros are.  Some use a total order maintained mostly by
hand, some use a partial ordering mechanism based on require/conflict/
provides.  I'm not talking theory, I'm talking a decade of practice.

 > OK.  What reasons do you have to NOT include the load-dir functionality
 > in the core if it's disabled by default and must be consciously enabled?
 > That you're happy without it?

No.  I maintain an Emacs, and I really don't want a feature that I am
sure will cause some users pain, but I intend to disclaim
responsibility for.  I've made that mistake a couple of times, and the
minor convenience it would provide to experienced users like you just
doesn't offset the user pain and maintainer embarrassment.

 > For me those reasons are modularization and easy code deployment on a
 > personal level.  I think I've been clear on that and wonder how you
 > determined your reasons were the good ones.

My definition of "good" is pretty strict here.  That is, "the need is
sufficiently great, and the problem sufficiently difficult for users
to solve on their own that it's appropriate to introduce new
problems."  The use of outline-minor-mode introduces no new problems
so the hurdle is low for it.  Since Emacs doesn't have per-function
autoloads, optional loading and autoloading clear a substantially
higher hurdle.  load-dir, OTOH, is easy to implement, and will be even
easier to install as a package.  "Sorry, you're not tall enough to ride
this ride."

 > Who said load-dir is supposed to avoid problems?  As with the earlier
 > comment about broken code, if you screw up your configuration layout in
 > any form, you go and fix it.  Emacs should indicate where the error
 > happened, but beyond that it's a user issue.

But here *Emacs is screwing up the configuration* by automatically
downloading and installing code, at least according to several
advocates' proposals.  Again, *this is not theory*.  This is based on
more than a decade of experience with such systems in XEmacs.  Users
do *not* view this as "oh, I messed up again".  They view it as "if
Emacs is not going to do it right, it should tell me to do it myself."

 > Sorry, I don't follow [the argument about non-idempotent configs].
 > Can you give a specific example?

Hooks written as lambdas, which will be added several times.

 > SJT> If Stefan and Yidong could get away with "Ted Z says that's
 > SJT> overengineering so we ain't gonna touch it", it might be worth
 > SJT> putting this in core.  But my recommendation to them is "snippet
 > SJT> conflicts are going to be a persistent annoyance in the long term;
 > SJT> let Ted Z and Jan D maintain a package and deal with the PEBKACs
 > SJT> and RFEs."
 > 
 > OK, got it.  Thanks.

You're welcome.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 12:20:53 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> For me those reasons are modularization and easy code deployment on a
>> personal level.  I think I've been clear on that and wonder how you
>> determined your reasons were the good ones.

SJT> My definition of "good" is pretty strict here.  That is, "the need is
SJT> sufficiently great, and the problem sufficiently difficult for users
SJT> to solve on their own that it's appropriate to introduce new
SJT> problems."

There's a lot of "daddy knows best" packed in your definition, isn't
there?  I find your "users will feel pain and the maintainer will suffer
embarassment" emotional arguments unappealing and somewhat patronizing,
personally.

>> Who said load-dir is supposed to avoid problems?  As with the earlier
>> comment about broken code, if you screw up your configuration layout in
>> any form, you go and fix it.  Emacs should indicate where the error
>> happened, but beyond that it's a user issue.

SJT> But here *Emacs is screwing up the configuration* by automatically
SJT> downloading and installing code, at least according to several
SJT> advocates' proposals.

I have not advocated such a thing and am against it; if I implied
otherwise I apologize for the misunderstanding.  el-get's *bootstrap*
may reside in the load-dir, but all the packages it manages won't.

SJT> Again, *this is not theory*.  This is based on more than a decade
SJT> of experience with such systems in XEmacs.  Users do *not* view
SJT> this as "oh, I messed up again".  They view it as "if Emacs is not
SJT> going to do it right, it should tell me to do it myself."

I view it as "I enabled the load-dir feature, maybe I should understand
it."  You keep insisting on holding the user's hand.

FWIW XEmacs rewrote and broke my .emacs without prompting last time I
had to use it to debug a Gnus problem, so I'm pessimistic about the way
it holds the user's hand.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > There's a lot of "daddy knows best" packed in your definition, isn't
 > there?

Of course.  That's what a maintainer does, make these kinds of
decisions about what's best for everybody.  That is the point of view
I'm taking, not trying to evaluate the feature on its own terms.

 > I have not advocated [automatically installing and loading code]
 > and am against it; if I implied otherwise I apologize for the
 > misunderstanding.  el-get's *bootstrap* may reside in the load-dir,
 > but all the packages it manages won't.

From the maintainer's point of view, this is *not about you*.  You
have restrained requirements for the feature, yes.  How do you propose
to impose *your* restraints on *others* who want additional features,
though?

 > SJT> Again, *this is not theory*.  This is based on more than a decade
 > SJT> of experience with such systems in XEmacs.  Users do *not* view
 > SJT> this as "oh, I messed up again".  They view it as "if Emacs is not
 > SJT> going to do it right, it should tell me to do it myself."
 > 
 > I view it as "I enabled the load-dir feature, maybe I should understand
 > it."

Again, how do you propose to get all other users to view it that way?
If this is only for you, you already know how to do it.  Obviously,
you expect this to be used by many people.  It needs to meet their
expectations as well as yours, or it's a bad idea to maintain it in
core.

It is my opinion, based on experience, that requests for more
automation will come up over and over again.  Similarly, requests for
support when code that users throw into the pot conflicts with code
already there will come up over and over again.  I don't think this
feature is worth inducing those requests, that's all.

 > You keep insisting on holding the user's hand.

No.  I'm anticipating what *some* users are going to expect of it,
based on experience with maintaining similar features.  I wouldn't
hold the user's hand in this case, I'd simply insist that the feature
be provided as a package.  Then helping them with their misconceptions
(and statistically speaking, there will be a certain fraction of users
who misunderstand) is my privilege as core maintainer, not my
responsibility.  Putting it in core makes it the other way around, and
that will suck for Stefan and Yidong IMO.

 > FWIW XEmacs rewrote and broke my .emacs without prompting last time I
 > had to use it to debug a Gnus problem, so I'm pessimistic about the way
 > it holds the user's hand.

Bingo!  That's exactly the result of the kind of feature creep I'm
worred about here, that I described as a mistake, which I now regret
not opposing more vehemently at the time, and which I am still not in
a position to unilaterally roll back.

And whatever happened to your "I enabled it, I should understand it"
view?  XEmacs was supposed to be a drop-in replacement for Emacs; you
shouldn't be surprised that it uses .emacs as its init file and its
custom-file if it finds it and custom-file isn't explicitly set, just
like Emacs does.  If you had spent the time to think about the
implications of what you were doing, I should think that "xemacs -q"[1]
would be the obvious thing to use to debug Gnus.  After all, the
person who reported the bug certainly wasn't using *your* .emacs in
*her* XEmacs.

But that's not the way you think about it, clearly.  And guess what?
You're right not to think about it that way, just as the users who
won't think so carefully about a trivial feature like load-dir are
right for them.  Life is too short.


Footnotes: 
[1]  Which causes both user-init-file and custom-file to be nil, so no
accesses, let alone writes, to .emacs would occur.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 10 Mar 2011 16:08:15 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> I have not advocated [automatically installing and loading code]
>> and am against it; if I implied otherwise I apologize for the
>> misunderstanding.  el-get's *bootstrap* may reside in the load-dir,
>> but all the packages it manages won't.

SJT> From the maintainer's point of view, this is *not about you*.  You
SJT> have restrained requirements for the feature, yes.  How do you propose
SJT> to impose *your* restraints on *others* who want additional features,
SJT> though?

By making the feature an optional, off-by-default loader and nothing more.

>> I view it as "I enabled the load-dir feature, maybe I should understand
>> it."

SJT> Again, how do you propose to get all other users to view it that way?
SJT> If this is only for you, you already know how to do it.  Obviously,
SJT> you expect this to be used by many people.  It needs to meet their
SJT> expectations as well as yours, or it's a bad idea to maintain it in
SJT> core.

I think you're arguing that the feature will be misinterpreted and
misused.  I disagree and can't find a way to agree with you; I still
think it's simple enough that it won't cause the kind of mismatched
expectations that lead to unwanted behavior and bug reports.

>> You keep insisting on holding the user's hand.

SJT> No.  I'm anticipating what *some* users are going to expect of it,
SJT> based on experience with maintaining similar features.  I wouldn't
SJT> hold the user's hand in this case, I'd simply insist that the feature
SJT> be provided as a package.  Then helping them with their misconceptions
SJT> (and statistically speaking, there will be a certain fraction of users
SJT> who misunderstand) is my privilege as core maintainer, not my
SJT> responsibility.  Putting it in core makes it the other way around, and
SJT> that will suck for Stefan and Yidong IMO.

While I disagree about the consequences you think it will have, I'm
proposing it before committing code exactly so the maintainers can make
up their mind.  I think we've both stated our case clearly at this point.

Ted

Re: user-controlled load-path extension: load-dir

[Stephen J. Turnbull]

Ted Zlatanov writes:
 
 > While I disagree about the consequences you think it will have, I'm
 > proposing it before committing code exactly so the maintainers can make
 > up their mind.  I think we've both stated our case clearly at this point.

Yup.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Wed, Mar 09, 2011 at 08:06:16AM +0100, Jan D. wrote:
> Ted Zlatanov skrev 2011-03-08 22:38:
> > On Tue, 08 Mar 2011 15:59:51 -0500 Stefan Monnier<monnier@IRO.UMontreal.CA>  wrote:
> >
> > SM>  If you're really talking about configuration code rather than packages,
> > SM>  then I tend to assume that people whose .emacs is so large as to need
> > SM>  modularization can figure out which kind of modularization they want and
> > SM>  implement (or copy&paste) the corresponding loop to load the various files.
> >
> > Yes, but you're assuming managing configuration modules in monolithic
> > Emacs Lisp is the best way.  Give us something simple and easy to manage
> > the loop at the filesystem level, so we don't have to write it
> > ourselves.
> >
> > SM>  I'm not even sure why you'd want to "modularize" in this way: my .emacs
> > SM>  was fairly large but splitting it into separate files never seemed like
> > SM>  a good way to help, since I'd then have to figure out how to make C-s
> > SM>  and M-/ find matches in neighboring files.  Instead I "split" it with
> > SM>  outline-minor-mode.
> >
> > For me it works better.  I like small files; outline-minor-mode and
> > folding-mode don't work for me.  I suspect I'm not the only one.  See
> > the URLs above for a list of similar needs.
> >
> 
> As others have said, I was also suprised that dropping a .el-fil in 
> ~/emacs.d didn't load it by default.  My emacs is quite big and I'd love 
> to split it up.  Another thing I really would like is for customize to 
> save its stuff in its own file and not in my .emacs.  With a load-dir 
> this would be trivial.
> 
> There is a real need an packages isn't it.  Its like wanting a rowing 
> boat and told to use the latest battle ship instead.

Can you take a look at Grail http://www.emacswiki.org/emacs/Grail and see if
that meets your needs ? If you want to modularize your configuration it is
explicity designed to do so.

> 
> 	Jan D.
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 0 bytes --]

Re: user-controlled load-path extension: load-dir

[Jan Djärv]

Mike Mattie skrev 2011-03-09 22.57:
> On Wed, Mar 09, 2011 at 08:06:16AM +0100, Jan D. wrote:

>>
>> There is a real need an packages isn't it.  Its like wanting a rowing
>> boat and told to use the latest battle ship instead.
>
> Can you take a look at Grail http://www.emacswiki.org/emacs/Grail and see if
> that meets your needs ? If you want to modularize your configuration it is
> explicity designed to do so.
>

Still not part of the core, and also too complicated.

	Jan D.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 4261 bytes --]

On Tue, Mar 08, 2011 at 04:26:32AM -0600, Ted Zlatanov wrote:
> On Mon, 07 Mar 2011 21:46:10 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> >> I think this proposal is really about code snippets that people would
> >> otherwise just cut and paste into their .emacs file.  The average
> >> user's .emacs often winds up containing mostly code they found
> >> somewhere and use without really understanding it.  Dropping each
> >> snippit in its own file would be a big help if the user ever did need
> >> to debug some problem with his init, or if he decided one day to
> >> actually learn elisp.
> 
> SM> I'm far from convinced it's better for people to drop random chunks of
> SM> configuration into separate files rather than all in the .emacs file.
> 
> It's not "better," it's what people already do.  Emacs would just make
> it easier than the current situation, which is to drop a file plus edit
> the .emacs every time a file is added or deleted.  What other choice is
> there to modularize .emacs?
> 
> SM> If you're talking about downloading random files containing
> SM> configuration code (so the user doesn't have to create a file, inventing
> SM> a file name for it), then maybe that could make sense, but I haven't
> SM> seen such files very much, and I suspect many of them could easily be
> SM> turned either into themes or into packages installable via package.el.
> 
> Obviously that works for package managers like el-get.
> 
> OK, let's say I have the 8 files I listed earlier (tzz.emacs.*.el).  I
> want to load them modularly.  So I put them in the load-dir.  Do I have
> to make 8 packages?  And every time I update one of those files, do I
> have to repackage it with a new version?  That seems workable but
> tedious compared to the code proposed by Ben Key and Evans Winner.
> 
> If you're against including the load-dir in the core and enabling it by
> default, how about making it a GNU ELPA package so it's easily
> installable?
> 
> On Tue, 8 Mar 2011 01:14:37 -0600 Ben Key <bkey76@gmail.com> wrote: 
> 
> BK> I am attaching to this message a quick implementation of this feature I
> BK> threw together in about an hour.  I am not familiar with the Emacs package
> BK> mechanism but I am certain that this could be converted to an Emacs package
> BK> without too much work thus making it easier to install.
> 
> On Tue, 08 Mar 2011 02:58:03 -0700 Evans Winner <ego111@gmail.com> wrote: 
> 
> EW> For what it's worth, this is the thing I have in my .emacs.
> EW> It does what I want it to, which is just to allow me to
> EW> break my .emacs into various not-excessively-long files
> EW> organized by subject, like email config, org-mode config and
> EW> so-forth.
> 
> Thanks for your code.  Both of your solutions are similar to what I
> would like to use, but Mike Mattie's concerns about startup behavior
> (mainly for daemon mode, for --batch mode, and when there are errors)
> are valid.  The recursion behavior should probably not be the default by
> parallel with other such systems I listed, e.g. /etc/rc.d and Bazaar's
> plugins directory.  It's easy enough to make it optional or to write a
> snippet that loads a specific subdirectory.  The argument I have against
> recursion is that it's hard to tell at a glance what's going to get
> loaded.

I have thought alot about your proposal for automatic loading so that it
is not necessary to edit anything, just drop a file in. There is some
real appeal for that kind of a feature ... but it is hard to implement.

By hard to implement I mean really hard. Just because a file is opened
and written to, it is not necessarily ready to load. I hit C-x s alot
when tweaking files ...

Even if I hacked some kind of inotify support to watch a directory for
IO activity I would never know when to actually load it.

If you can somehow signal when a collection is ready for re-scanning
I can see an implementation ...

> 
> It seems no one thinks prompting about new or changed files in the
> load-dir is worthwhile, so never mind about that part of my proposal.

I don't discount your idea, it's just hard to make a really reliable
implementation IMHO.

> 
> Thanks
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Jan D.]

Mike Mattie skrev 2011-03-09 07:03:

> I have thought alot about your proposal for automatic loading so that it
> is not necessary to edit anything, just drop a file in. There is some
> real appeal for that kind of a feature ... but it is hard to implement.
>
> By hard to implement I mean really hard. Just because a file is opened
> and written to, it is not necessarily ready to load. I hit C-x s alot
> when tweaking files ...
>
> Even if I hacked some kind of inotify support to watch a directory for
> IO activity I would never know when to actually load it.
>
> If you can somehow signal when a collection is ready for re-scanning
> I can see an implementation ...
>

It can't be hard, there are tons of programs that already have this 
feature.  Some have inotify style loading, other just load at startup. 
I would expect load at startup and then with a command (M-x 
load-load-dir or some such).

	Jan D.

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Wed, Mar 09, 2011 at 08:20:49AM +0100, Jan D. wrote:
> Mike Mattie skrev 2011-03-09 07:03:
> 
> > I have thought alot about your proposal for automatic loading so that it
> > is not necessary to edit anything, just drop a file in. There is some
> > real appeal for that kind of a feature ... but it is hard to implement.
> >
> > By hard to implement I mean really hard. Just because a file is opened
> > and written to, it is not necessarily ready to load. I hit C-x s alot
> > when tweaking files ...
> >
> > Even if I hacked some kind of inotify support to watch a directory for
> > IO activity I would never know when to actually load it.
> >
> > If you can somehow signal when a collection is ready for re-scanning
> > I can see an implementation ...
> >
> 
> It can't be hard, there are tons of programs that already have this 
> feature.  Some have inotify style loading, other just load at startup. 
> I would expect load at startup and then with a command (M-x 
> load-load-dir or some such).

That I can do.

> 
> 	Jan D.
> 
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 0 bytes --]

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 5562 bytes --]

On Mon, Mar 07, 2011 at 10:24:58AM -0600, Ted Zlatanov wrote:
> On Sat, 5 Mar 2011 11:11:34 -0800 Chad Brown <yandros@MIT.EDU> wrote: 
> 
> CB> On Mar 4, 2011, at 7:18 PM, Ted Zlatanov wrote:
> >> 
> >> By analogy consider some of the software that lets you put a snippet in
> >> a conf.d directory, obviously implying that this is convenient for the
> >> user.  This is just a sampling from my machine:
> 
> CB> Those are all machine-wide configurations.  For that, use site-lisp.
> 
> Bazaar has a plugins directory; files in it are automatically activated,
> as an example of a user-level facility like this.  Anyhow, my point is
> that placing a file in a directory is inherently more modular and
> convenient to the user than augmenting a single file.  Do you disagree?
> 
> CB> If the user has to go through a set of steps to install and activate a
> CB> package, how isn't that what ELPA does?
> 
> They are not packages, they are snippets of code.  ELPA requires far
> more structure and many more steps.  For what I'm proposing, a lot less
> work is required (just a y/n prompt the first time a snippet is found to
> ensure it's not placed there maliciously).
> 

This whole security thing is a real non-issue. There is no security in
Emacs, it (Emacs) simply uses the underlying system security
model. What is really at issue is whether the user _trusts_ the
code. Trying to address security issues in a software system that has
no security API/model leads to some skewed thinking, and I am not just
splitting hairs. Thinking in terms of trust leads to better reasoning.

> On Sat, 5 Mar 2011 23:21:51 -0800 Mike Mattie <codermattie@gmail.com> wrote: 
> 
> MM> There are ways to solve the problem you are looking at without wiring more
> MM> logic into Emacs. Use the .emacs file as a more sophisticated loader for a
> MM> complex configuration when necessary.
> 
> I can and do, sure.  I don't think it creates a nice user experience.
> 
> MM> I have done this with my Grail project (on EmacsWiki), as has tidyconfig.
> MM> I would look at these solutions first before proposing code that has to
> MM> be wired into the bootstrap codepath.
> 
> Thanks for pointing those out.  Would Grail or tidyconfig work as a core
> option on startup (meaning the user doesn't have to install a package,
> just customize a boolean)?  Do they handle signing the snippets the
> first time they are found?  Can you give us a comparison of them and any
> other similar solutions you know?  They have to be GPL and assigned to
> Emacs to be included.

I doubt Grail would be included in-tree. Maybe I am wrong, but I get that
vibe. I would certianly need to convert some recursive functions to iterative
versions before even bringing Grail to the table. I have been maturing Grail
out-of-tree to ensure the code is robust, designed well, and is targetted at
typical user needs.

> 
> MM> There are a number of issues a reliable loader should address:
> 
> MM> how does it handle errors ?
> 
> MM> how does it handle --batch ?
> 
> MM> how does it handle --daemon ?
> 
> MM> how simple,transparent,debuggable is the loader code ?
> 
> MM> I have not seen any code so far in your proposal unless I accidentally
> MM> skipped or deleted a message with it.
> 
> I can write the code but haven't seen the need for actual code yet (I
> could have just comitted some code to Emacs instead, but find that
> option to be antisocial).
> 
> I'd rather use something that already exists such as Grail or tidyconfig
> that you kindly pointed out.  If you could tell us how any potential
> solutions can help answer the questions above, that would be wonderful.
> I can look at any solutions and evaluate them if you like.

I can only speak for grail, but Grail essentially Handles the
load-path issue. When it comes to load-path Grail is
comprehensive. What grail does not do is automatically load code from
a given directory. Directories that grail looks for are added to
load-path when they exist. Grail looks for files such as keys.el
commands.el, user.el and does load those when found.  They are assumed
to be user created files. When snippets are collected in a directory
such as dist/elisp, or local/elisp, the user is expected to place a
require or similar command in one of these recognized files to
actually load the snippet/whatever at the start.

As far as error-handling goes grail does shine a bit in that errors within
a file are trapped, meaning that mistakes in one file does not cause the
entire configuration to abort leading to a --debug-init situtation. In this
regard Grail is extremely robust.

I will be updating the documentation of Grail extensively over the next couple
few days to describe it's use, usefulness, limitations etc .. I will send you
a mail off-list when I have articulated Grail properly.

> 
> To answer your questions from my perspective, IMHO a loader should work
> like package.el.  IOW, whenever and however package.el is loaded
> currently, this loader we're discussing should also be loaded, since
> it's effectively a package manager but with snippets instead of
> packages.

I am seriously considering how to add the kind of functionality you want to
Grail. I would like to add it to broaden the usefullness of Grail. I do not
like the idea of hacking another load-path 'ish variable into Emacs when simply
adding elisp to manage load-path is an option. Two similar variables is simply
bad design.

> 
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Mon, 7 Mar 2011 16:47:27 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> As far as error-handling goes grail does shine a bit in that errors within
MM> a file are trapped, meaning that mistakes in one file does not cause the
MM> entire configuration to abort leading to a --debug-init situtation. In this
MM> regard Grail is extremely robust.

That sounds useful indeed.

MM> I am seriously considering how to add the kind of functionality you want to
MM> Grail. I would like to add it to broaden the usefullness of Grail. I do not
MM> like the idea of hacking another load-path 'ish variable into Emacs when simply
MM> adding elisp to manage load-path is an option. Two similar variables is simply
MM> bad design.

Maybe you could provide separate packages: grail-loader, grail-load-dir
(the snippets directory), and grail-modules (the current Grail modules
like keys.el).  Then users can choose which layout they prefer and still
get the benefit of a safe loader.  Or perhaps grail-load-dir can simply
be part of grail-loader.

Ted

Re: user-controlled load-path extension: load-dir

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Tue, Mar 08, 2011 at 04:37:37AM -0600, Ted Zlatanov wrote:
> On Mon, 7 Mar 2011 16:47:27 -0800 Mike Mattie <codermattie@gmail.com> wrote: 
> 
> MM> As far as error-handling goes grail does shine a bit in that errors within
> MM> a file are trapped, meaning that mistakes in one file does not cause the
> MM> entire configuration to abort leading to a --debug-init situtation. In this
> MM> regard Grail is extremely robust.
> 
> That sounds useful indeed.
> 
> MM> I am seriously considering how to add the kind of functionality you want to
> MM> Grail. I would like to add it to broaden the usefullness of Grail. I do not
> MM> like the idea of hacking another load-path 'ish variable into Emacs when simply
> MM> adding elisp to manage load-path is an option. Two similar variables is simply
> MM> bad design.
> 
> Maybe you could provide separate packages: grail-loader, grail-load-dir
> (the snippets directory), and grail-modules (the current Grail modules
> like keys.el).  Then users can choose which layout they prefer and still
> get the benefit of a safe loader.  Or perhaps grail-load-dir can simply
> be part of grail-loader.

I have been reading this thread very carefully and have given the
snippets proposal alot of thought. I am willing to add a snippets
directory like functionality. I would require the user to execute a
command to signal to grail that the user has placed a new file in the
snippets directory.

BTW, Grail does allow you to change the layout, by creating a
grail-cfg.el file in USER_ELISP (see the Grail page for the meaning of
that variable). grail-cfg.el is loaded if it exists right after the
default tree is defined in defvar's and before the heavy lifting is done
in terms of loading.

> 
> Ted
> 
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Tue, 8 Mar 2011 22:26:01 -0800 Mike Mattie <codermattie@gmail.com> wrote: 

MM> I have been reading this thread very carefully and have given the
MM> snippets proposal alot of thought. I am willing to add a snippets
MM> directory like functionality. I would require the user to execute a
MM> command to signal to grail that the user has placed a new file in the
MM> snippets directory.

MM> BTW, Grail does allow you to change the layout, by creating a
MM> grail-cfg.el file in USER_ELISP (see the Grail page for the meaning of
MM> that variable). grail-cfg.el is loaded if it exists right after the
MM> default tree is defined in defvar's and before the heavy lifting is done
MM> in terms of loading.

I think Grail brings too much functionality for the proposal in this
discussion, after looking at its pieces.  Ben Key's implementation is
closer to the simplicity I was hoping for.  That's not to say Grail is
not useful, I just don't think that it is the right solution here.

Ted

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 898 bytes --]

Hello,

I personally do not think that this is a good idea for inclusion in Emacs.
However it is a fairly simple task to implement this feature in Emacs lisp.
I am attaching to this message a quick implementation of this feature I
threw together in about an hour.  I am not familiar with the Emacs package
mechanism but I am certain that this could be converted to an Emacs package
without too much work thus making it easier to install.

For now it can be installed by simply placing the file in a directory
located somewhere in the load path and adding the line
(require 'user-load-dir)
to your either .emacs or site-start.el file.

Then from that point on, it will be possible to do exactly what you want,
place a .el file in the directory specified by the user-load-dir variable
(~/.emacs.d/load by default) and have it loaded without having to make any
additional changes to your .emacs file.

[-- Attachment #1.2: Type: text/html, Size: 975 bytes --]

[-- Attachment #2: user-load-dir.el --]
[-- Type: application/octet-stream, Size: 1897 bytes --]

;;; user-load-dir.el -- A quick and dirty implementation of a user load
;;; directory.  This file is provided as a possible answer to the
;;; "user-controlled load-path extension: load-dir" discussion on the
;;; emacs-devel list found at
;;; <http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00048.html>.

;;; This could eventually be turned into an Emacs package that can be
;;; installed via package.el.  For now, this file can be installed by
;;; placing it somewhere in your load path and add the line
;;; (require 'user-load-dir)
;;; to your .emacs or site-start.el file.

;;; Copyright (c) 2011 Ben Key

;; This file is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This file is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with the bk-emacs-tools package.  If not, see
;; <http://www.gnu.org/licenses/>.

(defcustom user-load-dir
  (format "%s.emacs.d/load" (file-name-directory (expand-file-name user-init-file)))
  "A user defined directory in which the user can place .el files that
will automatically be loaded after Emacs initialization is complete."
  :type 'directory
  :group 'initialization
  :require 'user-load-dir
  )

(defun load-user-load-dir-files ()
  "Loads all .el files found in the `user-load-dir.'"
  (interactive)
  (mapc 'load (directory-files user-load-dir t "\\.el\\'"))
  )

(add-hook 'after-init-hook 'load-user-load-dir-files)

(provide 'user-load-dir)

Re: user-controlled load-path extension: load-dir

[Evans Winner]

For what it's worth, this is the thing I have in my .emacs.
It does what I want it to, which is just to allow me to
break my .emacs into various not-excessively-long files
organized by subject, like email config, org-mode config and
so-forth.  It's not very well tested, and I know that once
or twice I have gotten some kind of symlink to nowhere left
in that directory when a remote emacsclient session dies (at
least I think that's the cause) -- and then the result is a
file that ends in .el, but that can't be loaded and halts
the init process.  Anyway, for what it's worth.  This is the
sort of functionality I expected when I heard about the
~/.emacs.d/ concept, and was surprised when that's not what
it was.


(defun load-dir (directory &optional recurse)
  "Load all Emacs Lisp files in DIRECTORY.
If RECURSE is non-nil, recursively load all such files in all
subdirectories."
  (interactive "D")
  (dolist (file
	   (directory-files (expand-file-name directory) t
			    directory-files-no-dot-files-regexp) nil)
    (cond
     ((and recurse (file-directory-p file))
      (when recurse (load-dir file)))
     ((string-match "\\.el$\\|\\.elc$" file)
      (load-library (file-name-sans-extension file))))))

Re: user-controlled load-path extension: load-dir

[PJ Weisberg]

On Tue, Mar 8, 2011 at 1:58 AM, Evans Winner <ego111@gmail.com> wrote:

> the init process.  Anyway, for what it's worth.  This is the
> sort of functionality I expected when I heard about the
> ~/.emacs.d/ concept, and was surprised when that's not what
> it was.

I was also surprised that it wasn't possible to refactor my .emacs
file in this way.  Though it occurred to me after my last post that
the average non-programmer probably would never feel the need to
refactor his .emacs.

For what it's worth (approximately the cost of the electrons used for
this message), here's the code I added to the end of my .emacs to do
what I believe we're discussing:

======================================================================
(defvar init-directory "~/.emacs.d/init-el"
  "Directory in which PJ keeps elisp files to be executed on startup")

(push init-directory load-path)
(byte-recompile-directory init-directory 0)
(mapc 'load
      (mapcar (lambda(x) (substring x 0 -3))
              (directory-files init-directory nil ".*\\.el$")))
======================================================================

The part about byte-compiling the elisp files is totally unnecessary,
but it makes me feel better to think that things are slightly more
efficient than they could have been. ;-)

-PJ

bug#8205: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 1146 bytes --]

Hello,

I have attached to this message version 2 of user-load-dir.el.  This version
incorporates the recursive load-dir function written by Evans Winner (see <
http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>).  It
also adds two new defcustom variables,
load-files-in-user-load-dir-recursively and
auto-load-files-in-user-load-dir.

Note that one major new feature introduced by this version is that if the
.el file has been byte compiled, the byte compiled version will be loaded
instead of the source version.  This is a major improvement over my original
version which would only load .el files.

There are two remaining issues with this version.  If one of the files in
the user-load-dir has been byte-compiled, it will be loaded twice.  Also it
has no error handling.

I am more than willing to continue development of this file into an Emacs
package that can be included in ELPA.  This would make it possible for this
useful feature to be made readily available to those who want to use it
without including it in Emacs itself.

I have one question.  What sort of error handling should I incorporate into
this package?

[-- Attachment #1.2: Type: text/html, Size: 1306 bytes --]

[-- Attachment #2: user-load-dir.el --]
[-- Type: application/octet-stream, Size: 3741 bytes --]

;;; user-load-dir.el -- A quick and dirty implementation of a user load
;;; directory.  This file is provided as a possible answer to the
;;; "user-controlled load-path extension: load-dir" discussion on the
;;; emacs-devel list found at
;;; <http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00048.html>.

;;; This could eventually be turned into an Emacs package that can be
;;; installed via package.el.  For now, this file can be installed by
;;; placing it somewhere in your load path and add the line
;;; (require 'user-load-dir)
;;; to your .emacs or site-start.el file.

;;; Copyright (c) 2011 Ben Key

;; This file is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This file is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with the bk-emacs-tools package.  If not, see
;; <http://www.gnu.org/licenses/>.

(defcustom user-load-dir
  (format "%s.emacs.d/load" (file-name-directory (expand-file-name user-init-file)))
  "A user defined directory in which the user can place .el files that
will automatically be loaded after Emacs initialization is complete."
  :type 'directory
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom load-files-in-user-load-dir-recursively nil
  "Non-nil causes `load-user-load-dir-files' to load all .el or .elc
files in the `user-load-dir' and in all subdirectories of the
`user-load-dir.'  Nil causes `load-user-load-dir-files' to load only
the files in the `user-load-dir.'

The default value is nil."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom auto-load-files-in-user-load-dir t
"Non-nil causes `load-user-load-dir-files' to be called automatically
from the `after-init-hook.'  Nil suppresses this automatic loading.
Note that since `load-user-load-dir-files' is an interactive function,
you may call it manually by using 'M-x load-user-load-dir-files' if you
choose to set this variable to nil.

The default value is t."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defun load-dir (directory &optional recurse)
  "Load all Emacs Lisp files in DIRECTORY.
If RECURSE is non-nil, recursively load all such files in all
subdirectories.

This function was written by Evans Winner.  See the message
<http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>"
  (interactive "D")
  (dolist (file
           (directory-files (expand-file-name directory) t
                            directory-files-no-dot-files-regexp) nil)
    (cond
     ((and recurse (file-directory-p file))
      (when recurse (load-dir file)))
     ((string-match "\\.el$\\|\\.elc$" file)
      (load-library (file-name-sans-extension file))))))

(defun load-user-load-dir-files ()
  "Loads all .el files found in the `user-load-dir.'"
  (interactive)
  (load-dir user-load-dir load-files-in-user-load-dir-recursively)
  )

(defun user-load-dir-after-init-hook ()
  "An `after-init-hook' function that is used to conditionally call
`load-user-load-dir-files' depending on the value of
`auto-load-files-in-user-load-dir'."
  (if auto-load-files-in-user-load-dir
	  (load-user-load-dir-files)
	)
  )

(add-hook 'after-init-hook 'load-user-load-dir-files)

(provide 'user-load-dir)

RE: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 1146 bytes --]

Hello,

I have attached to this message version 2 of user-load-dir.el.  This version
incorporates the recursive load-dir function written by Evans Winner (see <
http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>).  It
also adds two new defcustom variables,
load-files-in-user-load-dir-recursively and
auto-load-files-in-user-load-dir.

Note that one major new feature introduced by this version is that if the
.el file has been byte compiled, the byte compiled version will be loaded
instead of the source version.  This is a major improvement over my original
version which would only load .el files.

There are two remaining issues with this version.  If one of the files in
the user-load-dir has been byte-compiled, it will be loaded twice.  Also it
has no error handling.

I am more than willing to continue development of this file into an Emacs
package that can be included in ELPA.  This would make it possible for this
useful feature to be made readily available to those who want to use it
without including it in Emacs itself.

I have one question.  What sort of error handling should I incorporate into
this package?

[-- Attachment #1.2: Type: text/html, Size: 1365 bytes --]

[-- Attachment #2: user-load-dir.el --]
[-- Type: application/octet-stream, Size: 3741 bytes --]

;;; user-load-dir.el -- A quick and dirty implementation of a user load
;;; directory.  This file is provided as a possible answer to the
;;; "user-controlled load-path extension: load-dir" discussion on the
;;; emacs-devel list found at
;;; <http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00048.html>.

;;; This could eventually be turned into an Emacs package that can be
;;; installed via package.el.  For now, this file can be installed by
;;; placing it somewhere in your load path and add the line
;;; (require 'user-load-dir)
;;; to your .emacs or site-start.el file.

;;; Copyright (c) 2011 Ben Key

;; This file is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This file is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with the bk-emacs-tools package.  If not, see
;; <http://www.gnu.org/licenses/>.

(defcustom user-load-dir
  (format "%s.emacs.d/load" (file-name-directory (expand-file-name user-init-file)))
  "A user defined directory in which the user can place .el files that
will automatically be loaded after Emacs initialization is complete."
  :type 'directory
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom load-files-in-user-load-dir-recursively nil
  "Non-nil causes `load-user-load-dir-files' to load all .el or .elc
files in the `user-load-dir' and in all subdirectories of the
`user-load-dir.'  Nil causes `load-user-load-dir-files' to load only
the files in the `user-load-dir.'

The default value is nil."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom auto-load-files-in-user-load-dir t
"Non-nil causes `load-user-load-dir-files' to be called automatically
from the `after-init-hook.'  Nil suppresses this automatic loading.
Note that since `load-user-load-dir-files' is an interactive function,
you may call it manually by using 'M-x load-user-load-dir-files' if you
choose to set this variable to nil.

The default value is t."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defun load-dir (directory &optional recurse)
  "Load all Emacs Lisp files in DIRECTORY.
If RECURSE is non-nil, recursively load all such files in all
subdirectories.

This function was written by Evans Winner.  See the message
<http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>"
  (interactive "D")
  (dolist (file
           (directory-files (expand-file-name directory) t
                            directory-files-no-dot-files-regexp) nil)
    (cond
     ((and recurse (file-directory-p file))
      (when recurse (load-dir file)))
     ((string-match "\\.el$\\|\\.elc$" file)
      (load-library (file-name-sans-extension file))))))

(defun load-user-load-dir-files ()
  "Loads all .el files found in the `user-load-dir.'"
  (interactive)
  (load-dir user-load-dir load-files-in-user-load-dir-recursively)
  )

(defun user-load-dir-after-init-hook ()
  "An `after-init-hook' function that is used to conditionally call
`load-user-load-dir-files' depending on the value of
`auto-load-files-in-user-load-dir'."
  (if auto-load-files-in-user-load-dir
	  (load-user-load-dir-files)
	)
  )

(add-hook 'after-init-hook 'load-user-load-dir-files)

(provide 'user-load-dir)

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ben Key <bkey76@gmail.com> writes: 
> I am more than willing to continue development of this file into 
> an Emacs package that can be included in ELPA.  This would make 
> it possible for this useful feature to be made readily available 
> to those who want to use it without including it in Emacs 
> itself. 
 
In my opinion, the biggest value by far of such a feature is to be 
able to depend on `user-load-dir' with no active collaboration of 
the user.  So please consider including in Emacs, not as an 
additional package.   Really the goal is to simplify as much as 
possible the steps required to install a new elisp file as a user. 
See how it goes each time (and that's only too often each day) on 
the help channel:  20:55 <dim> ,install
20:55 <fsbot> install -- [0] To install a <file>.el, save it to, 
say ~/elisp
20:55 <fsbot> [1] add to the beginning of ~/.emacs: (add-to-list 
'load-path 
              "~/elisp"), ..[Type ,more]
20:55 <dim> ,m
20:56 <fsbot> [2] now FOLLOW THE INSTALLATION INSTRUCTIONS IN THE 
FILE, 
              typically adding (require '<file>) to the end of 
              ~/.emacs,
20:56 <fsbot> [3] see 
http://www.emacswiki.org/cgi-bin/wiki.pl?LoadPath, ..[Type 
              ,more]
20:56 <dim> ,m
20:56 <fsbot> [4] the site-wide emacs extension directory is 
usually 
              `/usr/local/share/emacs/site-lisp',
20:56 <fsbot> [5] see autoload

With this new facility, we could say to newbies to just save the
<file>.el in ~/.emacs.d/load.d and do M-x load-user-files, or even M-x
load-new-user-files.  And of course we could bootstrap external scripts
much more easily: if we take el-get as an example, the user would never
have to worry about the load-path any more.  Ever.


Also I think that `user-load-dir' should default to "~/.emacs.d/load.d/".

I have no particular opinion on the autoloading facility that Julien
requires, because it seems to me that as soon as the `user-load-dir' is
there, it's easy to provide for a snippet that allow support for
"~/.emacs.d/autoload.d", whereas I don't see how to go the other route.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 8, 2011, at 12:01 PM, Dimitri Fontaine wrote:
> With this new facility, we could say to newbies to just save the
> <file>.el in ~/.emacs.d/load.d and do M-x load-user-files, or even M-x
> load-new-user-files.

Without the problems of a magic directory, we just just tell the user something
like:

``in emacs: M-x browse-packages RET, browse to <packagename>, and click `install'.''

or, for random `snippets' in things like github or emacswiki:

``in emacs: M-x install-by-url RET <paste URL here>''

and have emacs download the package, unpack it, describe it to the user, and offer to activate it (adding any necessary code to .emacs, probably via a custom-style equivalent).  This could also ask any required questions up front. This seems easier (for the user) and more capable (for the developer).  Why the opposition? Emacs already contains facilities that modify the user's startup files for them, and has for decades.

From the point of view of emacs 24 and inexperienced users, I again suggest that the Firefox extensions/themes mechanism is a much better model to examine than magic directories.
 
I hope that helps.
*Chad

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chad Brown <yandros@MIT.EDU> writes:
> ``in emacs: M-x install-by-url RET <paste URL here>''
>
> and have emacs download the package, unpack it, describe it to the user, and
> offer to activate it (adding any necessary code to .emacs, probably via a
> custom-style equivalent).

You are pretty much describing how el-get already works.

>  This could also ask any required questions up
> front. This seems easier (for the user) and more capable (for the
> developer).  Why the opposition? Emacs already contains facilities that
> modify the user's startup files for them, and has for decades.

I hate it when any other software edit my user files, that's the whole
point of `user-load-dir'.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Chad Brown]

On Mar 8, 2011, at 12:38 PM, Dimitri Fontaine wrote:

> You are pretty much describing how el-get already works.

Yep; that's why I suggested it to Ted back in the original thread. For emacs 24, I'd like to see the other parts added (the questions and configuration, basically). Package.el provides most of the building blocks for this already, which is why I suggested to Ted that he could perhaps usefully combine the two.

>> This could also ask any required questions up
>> front. This seems easier (for the user) and more capable (for the
>> developer).  Why the opposition? Emacs already contains facilities that
>> modify the user's startup files for them, and has for decades.
> 
> I hate it when any other software edit my user files, that's the whole
> point of `user-load-dir'.

I guess I can see your complaint, since I also don't like software munging my .emacs file, but I find it hard to believe that there are many (any?) users who need your simplified method and yet don't want code messing with .emacs automatically.  Take a look at the documentation for custom-file and see if that doesn't address your concern.

*Chad

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chad Brown <yandros@MIT.EDU> writes:
> I guess I can see your complaint, since I also don't like software munging
> my .emacs file, but I find it hard to believe that there are many (any?)
> users who need your simplified method and yet don't want code messing with
> .emacs automatically.

It's not that much about simplicity that it's about having the computer
do all the automatic work on its own.  `user-load-dir' would help with
that.

It could be that custom.el would help too, but I don't really see how.

The idea we're talking about here is to have a simpler way to enable a
piece of elisp code that you fetch from somewhere (mail attachment,
emacswiki, some blog, an automatic installer, etc).  I don't see how
much simpler we can go than "save the content in a <file>.el in
~/.emacs.d/load.d, then either restart Emacs or M-x reload-user-dir".
And for automatic installers, the location is known to be `user-load-dir'.

I see people that want the feature too, some have already written some
version of it, some are sending new implementation, so I think there's a
need to cover here.  That you don't share the need is fine, you just
won't use the feature, and maybe even explicitly

  (setq user-load-dir nil)

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 09 Mar 2011 10:36:59 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

DF> The idea we're talking about here is to have a simpler way to enable a
DF> piece of elisp code that you fetch from somewhere (mail attachment,
DF> emacswiki, some blog, an automatic installer, etc).  I don't see how
DF> much simpler we can go than "save the content in a <file>.el in
DF> ~/.emacs.d/load.d, then either restart Emacs or M-x reload-user-dir".
DF> And for automatic installers, the location is known to be `user-load-dir'.

DF> I see people that want the feature too, some have already written some
DF> version of it, some are sending new implementation, so I think there's a
DF> need to cover here.  That you don't share the need is fine, you just
DF> won't use the feature, and maybe even explicitly

DF>   (setq user-load-dir nil)

On Wed, 09 Mar 2011 11:01:38 +0100 Jan Djärv <jan.h.d@swipnet.se> wrote: 

JD> I can make an load-dir feature in .emacs too.  It is having this in
JD> emacs core, working without extra user work that is the point.

I think it's clear there's quite a few users who want the load-dir
feature for various reasons.  The URLs I posted show that it's been
needed for a while now although no one thought to add it to Emacs.
package.el and el-get will not do what we're asking for.  As proposed
it's disabled by default so it won't matter to the users who don't want it.

I'd like to know if the maintainers have a reason NOT to put
user-load-dir in the core.  If we can proceed in the core, I'll propose
a specific patch and we can discuss that.  Otherwise I'll make it a
package in the GNU ELPA which is not as convenient but still usable.

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

Hello,

What is the general consensus on this?  Should we build this feature in as
Ted Zlatanov has requested or should we simply continue the development of
the package I have started that implements this feature and perhaps make it
available on ELPA?

I fully understand the reasoning behind this request.  What Ted wants is to
be able to download a .el file from the Internet, place it into a given
directory, and not have to do anything else to make it just work.  He wants
an alternative to changing his .emacs file, either manually or through an
automated mechanism, that is simpler to use than the Emacs package
mechanism.

The user-load-dir.el file I have provided accomplishes this task.  I am more
than willing to continue developing it into an Emacs package (I just have
not read the documentation yet so I know little more about the Emacs package
mechanism than it exists).

I know that providing this functionality as an Emacs package does not do
precisely what Ted wants because it does not make it "just work" in a clean
Emacs installation, which I think is what he was really hoping for.  But it
does make it possible to make Emacs behave as he has requested after
installing a single Emacs package, which is the next best thing.

If we go forward with providing this feature as an Emacs Package, I will
need to either get permission from Evans Winner to include his load-dir
function or write an implementation of it myself.

What is the next step?

[-- Attachment #2: Type: text/html, Size: 1619 bytes --]

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 885 bytes --]

Hello,

I have attached to this message version 3 of user-load-dir.el.  This version
fixes a bug in the prior version in the usage of the after-init-hook.  It
also incorporates some basic error handling in the load-dir function.  The
load-dir function has also been modified so that when a .el file has been
byte-compiled, thus causing a .el and a .elc file of the same name to exist,
the file is only loaded once.  In addition, user-load-dir.el now works
correctly when it is loaded from site-start.el (I had to deal with the fact
that user-init-file is nil at that point).  Finally it now works correctly
in XEmacs (directory-files-no-dot-files-regexp is not defined in XEmacs).

I look forward to any feedback anyone might have to offer on this
implementation of the user-load-dir feature.  I also look forward to a
discussion about how this feature will be offered to Emacs users.

[-- Attachment #1.2: Type: text/html, Size: 932 bytes --]

[-- Attachment #2: user-load-dir.el --]
[-- Type: application/octet-stream, Size: 4843 bytes --]

;;; user-load-dir.el -- A quick and dirty implementation of a user load
;;; directory.  This file is provided as a possible answer to the
;;; "user-controlled load-path extension: load-dir" discussion on the
;;; emacs-devel list found at
;;; <http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00048.html>.

;;; This could eventually be turned into an Emacs package that can be
;;; installed via package.el.  For now, this file can be installed by
;;; placing it somewhere in your load path and add the line
;;; (require 'user-load-dir)
;;; to your .emacs or site-start.el file.

;;; Copyright (c) 2011 Ben Key

;; This file is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This file is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with the bk-emacs-tools package.  If not, see
;; <http://www.gnu.org/licenses/>.

(if (not (boundp 'directory-files-no-dot-files-regexp))
	(defconst directory-files-no-dot-files-regexp
	  "^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*"
	  "Regexp matching any file name except \".\" and \"..\".")
  )

(defcustom user-load-dir
  (if (not (null user-init-file))
	  ;; If this file is loaded from site-start.el, `user-init-file'
	  ;; will be nil, this will cause `file-name-directory' to throw an
	  ;; error due to the argument not being a string.  As a result, we
	  ;; only attempt to set the default value of `user-load-dir' here
	  ;; if `user-init-file' is not nil.
	  (format "%s.emacs.d/load.d" (file-name-directory (expand-file-name user-init-file)))
	nil
	)
  "A user defined directory in which the user can place .el files that
will automatically be loaded after Emacs initialization is complete."
  :type 'directory
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom load-files-in-user-load-dir-recursively nil
  "Non-nil causes `load-user-load-dir-files' to load all .el or .elc
files in the `user-load-dir' and in all subdirectories of the
`user-load-dir.'  Nil causes `load-user-load-dir-files' to load only
the files in the `user-load-dir.'

The default value is nil."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom auto-load-files-in-user-load-dir t
  "Non-nil causes `load-user-load-dir-files' to be called automatically
from the `after-init-hook.'  Nil suppresses this automatic loading.
Note that since `load-user-load-dir-files' is an interactive function,
you may call it manually by using 'M-x load-user-load-dir-files' if you
choose to set this variable to nil.

The default value is t."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defun load-dir (directory &optional recurse)
  "Load all Emacs Lisp files in DIRECTORY.
If RECURSE is non-nil, recursively load all such files in all
subdirectories.

This function was written by Evans Winner.  See the message
<http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>"
  (interactive "D")
  (let (this-file loaded-files-list)
	(dolist 
		(file 
		 (directory-files
		  (expand-file-name directory) 
		  t directory-files-no-dot-files-regexp) nil)
	  (cond
	   ((and recurse (file-directory-p file))
		(when recurse (load-dir file))
		)
	   ((string-match "\\.el$\\|\\.elc$" file)
		(setq this-file (file-name-sans-extension file))
		(if (not (member this-file loaded-files-list))
			(condition-case err
				(progn
				  (load-library this-file)
				  (setq loaded-files-list (add-to-list 'loaded-files-list this-file))
				  )
			  (message "An error occurred while loading '%s.'" file)
			  )
		  )
		)
	   )
	  )
	)
  )

(defun load-user-load-dir-files ()
  "Loads all .el files found in the `user-load-dir.'"
  (interactive)
  (load-dir user-load-dir load-files-in-user-load-dir-recursively)
  )

(defun user-load-dir-after-init-hook ()
  "An `after-init-hook' function that is used to conditionally call
`load-user-load-dir-files' depending on the value of
`auto-load-files-in-user-load-dir'."
  (if (null user-load-dir)
	  ;; If this file is loaded from site-start.el, `user-load-dir' may
	  ;; be nil at this point.  See the comments for `user-load-dir'
	  ;; above. If this happens, use the default value here.
	  (setq user-load-dir (format "%s.emacs.d/load.d" (file-name-directory (expand-file-name user-init-file))))
	)
  (if auto-load-files-in-user-load-dir
	  (load-user-load-dir-files)
	)
  )

(add-hook 'after-init-hook 'user-load-dir-after-init-hook)

(provide 'user-load-dir)

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 1587 bytes --]

Hello,

I have attached to this message version 4 of user-load-dir.el.

Several changes have been made to this version.

   1. The error handling in load-dir actually works as expected.  If one of
   the files in the user-load-dir generates an error, it is silently ignored
   and load-dir continues to load all the other files in the user-load-dir.
   2. Added the function byte-compile-dir to user-load-dir.el.  This
   function can be used to byte compile all files in the directory specified by
   the directory parameter.  If the optional parameter recurs is non-nil, this
   is done recursively.
   3. Added the function byte-compile-user-load-dir-files.  This interactive
   function may be used to byte compile all files in the user-load-dir.
   4. The function load-user-load-dir-files does nothing if user-load-dir is
   nil.
   5. Modified the user-load-dir defcustom so that it is set to t instead of
   nil in the case that it cannot be initialized at load time due to
   user-init-file being nil.
   6. Modified user-load-dir-after-init-hook so that it only sets
   user-load-dir to its default value if it is equal to t.

The reason I made changes 5 and 6 is to allow for the possibility of this
feature being disabled by the user setting user-load-dir to nil.

Setting auto-load-files-in-user-load-dir to nil causes to not be called
automatically by the user-load-dir-after-init-hook but the files can still
be called by calling load-user-load-dir-files interactively.  Setting
user-load-dir
to nil disables load-user-load-dir-files and
byte-compile-user-load-dir-files.

[-- Attachment #1.2: Type: text/html, Size: 1912 bytes --]

[-- Attachment #2: user-load-dir.el --]
[-- Type: application/octet-stream, Size: 5588 bytes --]

;;; user-load-dir.el -- A quick and dirty implementation of a user load
;;; directory.  This file is provided as a possible answer to the
;;; "user-controlled load-path extension: load-dir" discussion on the
;;; emacs-devel list found at
;;; <http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00048.html>.

;;; This could eventually be turned into an Emacs package that can be
;;; installed via package.el.  For now, this file can be installed by
;;; placing it somewhere in your load path and add the line
;;; (require 'user-load-dir)
;;; to your .emacs or site-start.el file.

;;; Copyright (c) 2011 Ben Key

;; This file is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This file is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with the bk-emacs-tools package.  If not, see
;; <http://www.gnu.org/licenses/>.

(if (not (boundp 'directory-files-no-dot-files-regexp))
	(defconst directory-files-no-dot-files-regexp
	  "^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*"
	  "Regexp matching any file name except \".\" and \"..\".")
  )

(defcustom user-load-dir
  (if (not (null user-init-file))
	  ;; If this file is loaded from site-start.el, `user-init-file'
	  ;; will be nil, this will cause `file-name-directory' to throw an
	  ;; error due to the argument not being a string.  As a result, we
	  ;; only attempt to set the default value of `user-load-dir' here
	  ;; if `user-init-file' is not nil.
	  (format "%s.emacs.d/load.d" (file-name-directory (expand-file-name user-init-file)))
	t
	)
  "A user defined directory in which the user can place .el files that
will automatically be loaded after Emacs initialization is complete."
  :type 'directory
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom load-files-in-user-load-dir-recursively nil
  "Non-nil causes `load-user-load-dir-files' to load all .el or .elc
files in the `user-load-dir' and in all subdirectories of the
`user-load-dir.'  Nil causes `load-user-load-dir-files' to load only
the files in the `user-load-dir.'

The default value is nil."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defcustom auto-load-files-in-user-load-dir t
  "Non-nil causes `load-user-load-dir-files' to be called automatically
from the `after-init-hook.'  Nil suppresses this automatic loading.
Note that since `load-user-load-dir-files' is an interactive function,
you may call it manually by using 'M-x load-user-load-dir-files' if you
choose to set this variable to nil.

The default value is t."
  :type 'boolean
  :group 'initialization
  :require 'user-load-dir
  )

(defun load-dir (directory &optional recurse)
  "Load all Emacs Lisp files in DIRECTORY.
If RECURSE is non-nil, recursively load all such files in all
subdirectories.

This function is based on one written by Evans Winner.  See the message
<http://lists.gnu.org/archive/html/emacs-devel/2011-03/msg00239.html>
for details."
  (interactive "D")
  (let (this-file loaded-files-list)
	(dolist 
		(file 
		 (directory-files
		  (expand-file-name directory) 
		  t directory-files-no-dot-files-regexp) nil)
	  (cond
	   ((and recurse (file-directory-p file))
		(when recurse (load-dir file))
		)
	   ((string-match "\\.el$\\|\\.elc$" file)
		(setq this-file (file-name-sans-extension file))
		(if (not (member this-file loaded-files-list))
			(progn
			  (setq loaded-files-list (add-to-list 'loaded-files-list this-file))
			  (ignore-errors
				(load this-file))
			  )
		  )
		)
	   )
	  )
	)
  )

(defun load-user-load-dir-files ()
  "Loads all .el files found in the `user-load-dir.'"
  (interactive)
  (if (not (null user-load-dir))
	  (load-dir user-load-dir load-files-in-user-load-dir-recursively)
	)
  )

(defun byte-compile-dir (directory &optional recurse)
  "Byte compile all Emacs Lisp files in DIRECTORY.  If RECURSE is
non-nil, recursively byte compile all such files in all
subdirectories."
  (interactive "D")
  (dolist 
	  (file 
	   (directory-files
		(expand-file-name directory) 
		t directory-files-no-dot-files-regexp) nil)
	(cond
	 ((and recurse (file-directory-p file))
	  (when recurse (byte-compile-dir file))
	  )
	 ((string-match "\\.el$" file)
	  (ignore-errors
		(byte-compile-file file))
	  )
	 )
	)
  )

(defun byte-compile-user-load-dir-files ()
  "Byte compiles all .el files found in the `user-load-dir.'"
  (interactive)
  (if (not (null user-load-dir))
	  (byte-compile-dir user-load-dir load-files-in-user-load-dir-recursively)
	)
  )

(defun user-load-dir-after-init-hook ()
  "An `after-init-hook' function that is used to conditionally call
`load-user-load-dir-files' depending on the value of
`auto-load-files-in-user-load-dir'."
  (if (equal user-load-dir t)
	  ;; If this file is loaded from site-start.el, `user-load-dir' may
	  ;; be nil at this point.  See the comments for `user-load-dir'
	  ;; above. If this happens, use the default value here.
	  (setq user-load-dir
			(format "%s.emacs.d/load.d" (file-name-directory (expand-file-name user-init-file))))
	)
  (if auto-load-files-in-user-load-dir
	  (load-user-load-dir-files)
	)
  )

(add-hook 'after-init-hook 'user-load-dir-after-init-hook)

(provide 'user-load-dir)

Re: user-controlled load-path extension: load-dir

[Jan D.]

Hello.

What is wrong with (expand-file-name "~/emacs.d/load.d")?  I don't think 
user-init-file can be nil either.

It is much better to explicitly check for "." and ".." than use 
directory-files-no-dot-files-regexp.  That regexp is really really slow.

	Jan D.


Ben Key skrev 2011-03-09 07:50:
> Hello,
>
> I have attached to this message version 4 of user-load-dir.el.
>
> Several changes have been made to this version.
>
>    1. The error handling in load-dir actually works as expected.  If one
>       of the files in the user-load-dir generates an error, it is
>       silently ignored and load-dir continues to load all the other
>       files in the user-load-dir.
>    2. Added the function byte-compile-dir to user-load-dir.el.  This
>       function can be used to byte compile all files in the directory
>       specified by the directory parameter.  If the optional parameter
>       recurs is non-nil, this is done recursively.
>    3. Added the function byte-compile-user-load-dir-files.  This
>       interactive function may be used to byte compile all files in the
>       user-load-dir.
>    4. The function load-user-load-dir-files does nothing if
>       user-load-dir is nil.
>    5. Modified the user-load-dir defcustom so that it is set to t
>       instead of nil in the case that it cannot be initialized at load
>       time due to user-init-file being nil.
>    6. Modified user-load-dir-after-init-hook so that it only sets
>       user-load-dir to its default value if it is equal to t.
>
> The reason I made changes 5 and 6 is to allow for the possibility of
> this feature being disabled by the user setting user-load-dir to nil.
>
> Setting auto-load-files-in-user-load-dir to nil causes to not be called
> automatically by the user-load-dir-after-init-hook but the files can
> still be called by calling load-user-load-dir-files interactively.
> Setting user-load-dir to nil disables load-user-load-dir-files and
> byte-compile-user-load-dir-files.
>

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Wed, Mar 9, 2011 at 1:29 AM, Jan D. <jan.h.d@swipnet.se> wrote:

> What is wrong with (expand-file-name "~/emacs.d/load.d")?
> I don't think user-init-file can be nil either.

As I said in my comments for the user-load-dir defcustom, "If this file is
loaded from site-start.el, `user-init-file' will be nil."  Apparently
site-start.el is loaded before the code in lread.c that initializes the
Vuser_init_file variable.  I was surprised myself when I made the change to
load user-load-dir.el from my site-start.el and I got an error on the
defcustom line.

As for expanding "~/emacs.d/load.d," it may have the same problem with being
called from site-start.el.  If it does not, I will use it.

[-- Attachment #2: Type: text/html, Size: 846 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 9 Mar 2011 00:50:23 -0600 Ben Key <bkey76@gmail.com> wrote: 

BK> ;;; user-load-dir.el -- A quick and dirty implementation of a user load
BK> ;;; directory.

I agree with Jan that `directory-files-no-dot-files-regexp' is not
necessary.

`user-load-dir' should be `user-load-dirs' and customizable as a list of
directory names, not a single directory.

The variables and functions in user-load-dir.el should start with
"user-load-dir".  For example `load-files-in-user-load-dir-recursively'
is not a good name.  If this is in the core the variable names could be
different but in a library they shouldn't be.  I suggest several name
fixes below but it needs to be fixed everywhere.

`load-files-in-user-load-dir-recursively' would be better named
`user-load-dir-recursive'.

`auto-load-files-in-user-load-dir' is not needed.  If `user-load-dirs'
is not empty that implies you want to load the things in it.

`byte-compile-dir' should be named `user-load-dir-byte-compile'.
Similarly `byte-compile-user-load-dir-files' should be
`user-load-dir-byte-compile-files'.

It would also be good to make sure the code works with Emacs 23.1
through 23.3 so users can take it there as well.

Ted

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Ben Key <bkey76@gmail.com> writes:

> What is the general consensus on this?  Should we build this feature
> in as Ted Zlatanov has requested or should we simply continue the
> development of the package I have started that implements this feature
> and perhaps make it available on ELPA?

I still don't understand what "this" is.  Why do you need a full-blown
package to implement

(dolist (f (directory-files dir t "\\.el$"))
  (load foo))

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

To begin with using

(dolist (f (directory-files dir t "\\.el$"))
 (load foo))

does not support error handling so that Emacs will barf if one of the .el
files has an error in it.  Also, it does not support loading of files
recursively or loading byte-compiled files (it will load from source even if
byte compiled files are present).

Also, while this problem could be solved by adding those few lines to
,emacs, users would need to define dir themselves.  This package is intended
partially to allow someone who is not familiar with ELisp to just drop a
file they downloaded into a directory and have Emacs load it.

The benefit of providing a package to implement this functionality is that
the user of the package does not need to figure out how to define dir
themselves, it provides error handling so that Emacs initialization will not
be interrupted while loading the files in the load dir, it optionally
supports loading files recursively, and supports loading of byte compiled
files.  To support all of that your two lines quickly grows to be
considerably more.  My current implementation has about 100 lines if you
strip out the comments.  That is why a package is required, to provide
features your quick and dirty two line solution does not.

[-- Attachment #2: Type: text/html, Size: 1334 bytes --]

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 09 Mar 2011 14:51:22 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ben Key <bkey76@gmail.com> writes:
>> What is the general consensus on this?  Should we build this feature
>> in as Ted Zlatanov has requested or should we simply continue the
>> development of the package I have started that implements this feature
>> and perhaps make it available on ELPA?

CY> I still don't understand what "this" is.  Why do you need a full-blown
CY> package to implement

CY> (dolist (f (directory-files dir t "\\.el$"))
CY>   (load foo))

That's essentially it, but we want to be able to turn it on by
customizing a variable to be a LIST of directories (defaulting to nil),
not adding code to .emacs.  All the other improvements in Ben Key's
package, including byte-compilation and recursion, are nice-to-have but
not essential to our purpose.

Ted

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> That's essentially it, but we want to be able to turn it on by
> customizing a variable to be a LIST of directories (defaulting to
> nil), not adding code to .emacs.  All the other improvements in Ben
> Key's package, including byte-compilation and recursion, are
> nice-to-have but not essential to our purpose.

OK, I guess there's nothing wrong with adding a function to load
everything in a directory.  But what's the difference between having a
variable that defaults to nil and just letting users add

  (load-directory "~/.emacs.d/my-stuff")

to their init file?

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 09 Mar 2011 15:32:08 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> That's essentially it, but we want to be able to turn it on by
>> customizing a variable to be a LIST of directories (defaulting to
>> nil), not adding code to .emacs.  All the other improvements in Ben
>> Key's package, including byte-compilation and recursion, are
>> nice-to-have but not essential to our purpose.

CY> OK, I guess there's nothing wrong with adding a function to load
CY> everything in a directory.  But what's the difference between having a
CY> variable that defaults to nil and just letting users add

CY>   (load-directory "~/.emacs.d/my-stuff")

CY> to their init file?

The former is easier to customize and requires no extra code to be
written.  For the el-get installer in particular it is easier and less
intrusive to customize a variable than to insert code in the .emacs.
But I am not strongly opposed to the latter.

Ted

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chong Yidong <cyd@stupidchicken.com> writes:
> OK, I guess there's nothing wrong with adding a function to load
> everything in a directory.  But what's the difference between having a
> variable that defaults to nil and just letting users add
>
>   (load-directory "~/.emacs.d/my-stuff")
>
> to their init file?

The difference is that I'm after a solution where it's possible to
achieve that with *no* editing of the user-init-file.  At all.  So that
my el-get installer will just drop a file in `user-load-dir`.

The alternative is editing user-init-file to change the load-path and
(require 'el-get), or to explicitly (load "path/to/el-get/el-get.el"),
and that's not user friendly.

Please note that my requirement would not be fulfilled at all should the
load-directory be in a package.  What I need is to determine from code
where to place some file that will be loaded at emacs startup
automatically.  If the feature is not in code, that means I depend on
manual actions from users before to be able to have a working solution.

That would only be moving the problem, not solving it.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

>>>>> "Dimitri" == Dimitri Fontaine <dim@tapoueh.org> writes:

Dimitri> The difference is that I'm after a solution where it's possible to
Dimitri> achieve that with *no* editing of the user-init-file.  At all.  So that
Dimitri> my el-get installer will just drop a file in `user-load-dir`.

Your use case is completely solved by package.el.  First, package el-get
as a package (trivial).  Second, have an autoload cookie in el-get that
loads the code from some directory.

Why do you think you need anything more?

Tom

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Tom Tromey <tromey@redhat.com> writes:
> Your use case is completely solved by package.el.  First, package el-get
> as a package (trivial).  Second, have an autoload cookie in el-get that
> loads the code from some directory.
>
> Why do you think you need anything more?

I'm thinking I need much less than all that.

Then I've been trying to follow your advice and turn el-get into a
proper Emacs package, as that's easy enough, right?  Turns out not to
be.

Given the following (or some variations, all based upon reading (info
"(elisp) Packaging Basics") and (info "(elisp) Library Headers"), then
the describe-function of `version-to-list' and helpers):

;; Version: 1.2~dev
;; Package-Version: 1.2+alpha1

All I get from `package-install-from-buffer' is: Package does not define
a usable "Version" or "Package-Version" header.

It seem to me that the function `package-strip-rcs-id' is buggy from
failing to use `version-to-list' and instead invents its own regexp to
match the version identifier, then returns nil which gets into an error.

I'm still thinking that a simple directory where to put files to be
loaded would be a good feature, independently of how I will fix the
el-get-installer automatic bootstrap.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

>>>>> "Dimitri" == Dimitri Fontaine <dim@tapoueh.org> writes:

Dimitri> Given the following (or some variations, all based upon reading (info
Dimitri> "(elisp) Packaging Basics") and (info "(elisp) Library Headers"), then
Dimitri> the describe-function of `version-to-list' and helpers):

Dimitri> ;; Version: 1.2~dev
Dimitri> ;; Package-Version: 1.2+alpha1

Weird version number.

Dimitri> It seem to me that the function `package-strip-rcs-id' is buggy
Dimitri> from failing to use `version-to-list' and instead invents its
Dimitri> own regexp to match the version identifier, then returns nil
Dimitri> which gets into an error.

If version-to-list accepts that version but package.el still gives an
error, then it is a bug.  Please report it to the bug tracker, thanks.

Tom

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Tom Tromey <tromey@redhat.com> writes:

> Dimitri> ;; Version: 1.2~dev
> Dimitri> ;; Package-Version: 1.2+alpha1
>
> Weird version number.

Indeed, neither of these version strings are acceptable to
`version-to-list' (see its docstring).

Re: user-controlled load-path extension: load-dir

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Thu, 10 Mar 2011 13:56:55 -0500
> Cc: Ted Zlatanov <tzz@lifelogs.com>, Dimitri Fontaine <dim@tapoueh.org>,
> 	emacs-devel@gnu.org
> 
> Tom Tromey <tromey@redhat.com> writes:
> 
> > Dimitri> ;; Version: 1.2~dev
> > Dimitri> ;; Package-Version: 1.2+alpha1
> >
> > Weird version number.
> 
> Indeed, neither of these version strings are acceptable to
> `version-to-list' (see its docstring).

??

  (version-to-list "1.2+alpha1") => (1 2 -3 1)

This is in Emacs 23.3.

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chong Yidong <cyd@stupidchicken.com> writes:

> Tom Tromey <tromey@redhat.com> writes:
>
>> Dimitri> ;; Version: 1.2~dev
>> Dimitri> ;; Package-Version: 1.2+alpha1
>>
>> Weird version number.
>
> Indeed, neither of these version strings are acceptable to
> `version-to-list' (see its docstring).

ELISP> (version-to-list "1.2~dev")
*** Eval error ***  Invalid version syntax: '1.2~dev'

ELISP> (version-to-list "1.2+alpha")
(1 2 -3)

ELISP> (version-to-list "1.2+alpha1")
(1 2 -3 1)

ELISP> (version-to-list "1.2+alpha1")
(1 2 -3 1)

-- 
dim

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Dimitri Fontaine <dim@tapoueh.org> writes:

>> Indeed, neither of these version strings are acceptable to
>> `version-to-list' (see its docstring).
>
> ELISP> (version-to-list "1.2+alpha")
> (1 2 -3)

My mistake.  This is, indeed, a bug in package-strip-rcs-id.  I've just
committed a fix, thanks.

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Chong Yidong <cyd@stupidchicken.com> writes:
>> ELISP> (version-to-list "1.2+alpha")
>> (1 2 -3)
>
> My mistake.  This is, indeed, a bug in package-strip-rcs-id.  I've just
> committed a fix, thanks.

Thanks, will test later this week-end!
-- 
dim

Re: user-controlled load-path extension: load-dir

[Jambunathan K]

> If version-to-list accepts that version but package.el still gives an
> error, then it is a bug.  Please report it to the bug tracker, thanks.

This is out of thread ...

In package-tar-file-info, the following regexp is used for validating
the package name.

  (unless (string-match "^\\(.+\\)-\\([0-9.]+\\)\\.tar$" file)
    (error "Invalid package name `%s'" file))

The above regexp wouldn't catch files with the following version strings
1.0pre2 1.0.7.5 22.8beta3 0.9alpha1 6.9.30Beta.

Furthermore, if I do build a org-7.5pre1.tar, then list-packages gets
very confused about the name.

I believe some sort of auditing around numeric <-> stringified versions
of package files could be taken up.

Jambunathan K.

>
> Tom

Re: user-controlled load-path extension: load-dir

[chad]

On Mar 10, 2011, at 6:30 AM, Dimitri Fontaine wrote:
> 
> The difference is that I'm after a solution where it's possible to
> achieve that with *no* editing of the user-init-file.  At all.  So that
> my el-get installer will just drop a file in `user-load-dir`.

I don't understand your resistance to having the installer do the editing
instead of the user. This seems like a more general solution that avoids
some potential problems and would be useful in situations where a
magic directory would not (for example, where *some* configuration
is required but easily answered by the user).

The end result seems at least as user-friendly, except perhaps in a few
degenerate cases. What am I missing?

*Chad

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

chad <yandros@gmail.com> writes:
> I don't understand your resistance to having the installer do the editing
> instead of the user. This seems like a more general solution that avoids
> some potential problems and would be useful in situations where a
> magic directory would not (for example, where *some* configuration
> is required but easily answered by the user).

I just frown at the idea that a *user* file is edited by anyone else
than the user, explicitly.

> The end result seems at least as user-friendly, except perhaps in a few
> degenerate cases. What am I missing?

It's not user friendly to go edit the user file from a program in my
book.  Like not at all.

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[chad]

On Mar 11, 2011, at 12:41 PM, Dimitri Fontaine wrote:

> chad <yandros@gmail.com> writes:
>> I don't understand your resistance to having the installer do the editing
>> instead of the user. This seems like a more general solution that avoids
>> some potential problems and would be useful in situations where a
>> magic directory would not (for example, where *some* configuration
>> is required but easily answered by the user).
> 
> I just frown at the idea that a *user* file is edited by anyone else
> than the user, explicitly.

Emacs' configuration file has been edited automatically by emacs since 
at least version 18.43 (the first version I used regularly).  Start emacs 
without a .emacs file and hit `C-x n n y' and see what happens.

>> The end result seems at least as user-friendly, except perhaps in a few
>> degenerate cases. What am I missing?
> 
> It's not user friendly to go edit the user file from a program in my
> book.  Like not at all.

Have you ever run any Gnome or KDE programs? I think your expectations 
of `user friendly' might not match either new or experienced emacs users.

This conversation is becoming difficult for me to continue (I'm traveling),
and it doesn't seem like it's going anywhere, so I'm going to bow out now.
Thanks for working on el-get, and I hope that it all works out in the end.

*Chad

Re: user-controlled load-path extension: load-dir

[Jan Djärv]

12 mar 2011 kl. 06:51 skrev chad <yandros@gmail.com>:

> On Mar 11, 2011, at 12:41 PM, Dimitri Fontaine wrote:
> 
>> 
>> It's not user friendly to go edit the user file from a program in my
>> book.  Like not at all.
> 
> Have you ever run any Gnome or KDE programs? I think your expectations 
> of `user friendly' might not match either new or experienced emacs users.
> 

No Gnome or KDE program I know of edits user configuration files. They edit the programs private configuration file, but these aren't user configuration files
that the user is supposed to edit.

The user configures the program with a GUI interface. This is not the same as with Emacs where the user is explicitly told to edit .emacs.

     Jan D.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 11 Mar 2011 10:00:14 -0500 chad <yandros@gmail.com> wrote: 

c> a magic directory

There's nothing magic about the load-dir.  It's a list of directory
names, just like the load path.  Any .el files in those directories get
loaded on startup and when the user requests it explicitly.

There may be some extra rules to do with .elc files and recursion, but
both the idea and the implementation fit in a paragraph and are so
simple that calling them "magic" is misleading.

Ted

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 09 Mar 2011 15:32:08 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> That's essentially it, but we want to be able to turn it on by
>> customizing a variable to be a LIST of directories (defaulting to
>> nil), not adding code to .emacs.  All the other improvements in Ben
>> Key's package, including byte-compilation and recursion, are
>> nice-to-have but not essential to our purpose.

CY> OK, I guess there's nothing wrong with adding a function to load
CY> everything in a directory.  But what's the difference between having a
CY> variable that defaults to nil and just letting users add

CY>   (load-directory "~/.emacs.d/my-stuff")

CY> to their init file?

Chong, Stefan, can I take this and the lack of new posts in this thread
to mean it's OK to commit an experimental implementation of the load-dir
idea?  I don't know if Ben Key's paperwork is in progress but as you
observed it's not terribly complicated code so I can implement it in a
functionally similar way quickly (adding his work once we have his
copyright assignment).

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> Chong, Stefan, can I take this and the lack of new posts in this thread
> to mean it's OK to commit an experimental implementation of the load-dir
> idea?

Adding a load-directory function is OK, yes.  But please send the
corresponding patch for review first.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Wed, 16 Mar 2011 11:43:09 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Chong, Stefan, can I take this and the lack of new posts in this thread
>> to mean it's OK to commit an experimental implementation of the load-dir
>> idea?

SM> Adding a load-directory function is OK, yes.  But please send the
SM> corresponding patch for review first.

First, the library.  I'm not sure about the name (`user-load-dirs-load'
sounds terrible) and I've omitted the byte-compilation stuff that was in
Ben Key's code.  It's a full rewrite of his code, which makes copyright
assignments unnecessary so we can speed up the process.

Do we want to support loading byte-compiled files?

Is the messaging too intrusive?

Also I don't trap errors.  I think that's the right approach.  These are
not packages, they are like a part of your .emacs and should be treated
accordingly.  This is also why I'm sort of avoiding the byte compilation
support, although it's pretty easy to add.

Thanks
Ted


[-- Attachment #2: user-load-dirs.el --]
[-- Type: application/emacs-lisp, Size: 3439 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>>> Chong, Stefan, can I take this and the lack of new posts in this thread
>>> to mean it's OK to commit an experimental implementation of the load-dir
>>> idea?
SM> Adding a load-directory function is OK, yes.  But please send the
SM> corresponding patch for review first.

> First, the library.

Sounds like this is a package, which I'd be happy to see in ELPA, but
in core.


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 16 Mar 2011 22:13:24 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> Chong, Stefan, can I take this and the lack of new posts in this thread
>>>> to mean it's OK to commit an experimental implementation of the load-dir
>>>> idea?
SM> Adding a load-directory function is OK, yes.  But please send the
SM> corresponding patch for review first.

>> First, the library.

SM> Sounds like this is a package, which I'd be happy to see in ELPA, but
SM> in core.

I wrote it as a library so all we have to do in the core is call
`user-load-dirs-load' (or some alias to it) after loading the
`user-init-file'.  The central point several of us have been making is
that we want `user-load-dirs-load' called automatically, not when the
package or library are explicitly loaded, to make using this library a
simple matter of customizing a single variable (the
`user-load-dirs-list').

If you and Chong don't want that, then I'll put it in the ELPA.

Ted

Re: user-controlled load-path extension: load-dir

[PJ Weisberg]

2011/3/16 Ted Zlatanov <tzz@lifelogs.com>:
> On Wed, 16 Mar 2011 22:13:24 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>>>>> Chong, Stefan, can I take this and the lack of new posts in this thread
>>>>> to mean it's OK to commit an experimental implementation of the load-dir
>>>>> idea?
> SM> Adding a load-directory function is OK, yes.  But please send the
> SM> corresponding patch for review first.
>
>>> First, the library.
>
> SM> Sounds like this is a package, which I'd be happy to see in ELPA, but
> SM> in core.
>
...
>
> If you and Chong don't want that, then I'll put it in the ELPA.

I only saw Stefan say to go ahead with a load-directory function,
which I assumed would live in files.el next to load-file and look
something like this:

(defun load-directory (dir recursive)
 "Load all Lisp files in DIR.  If RECURSIVE is non-nil,
recursively call load-directory for all subdirectories of dir."
 (interactive "DLoad Dir: \nP")
 (dolist (file (directory-files dir t ".*\\.el$"))
   ;; Cut off the .el so load will pick up the .elc version if it's there
   (setq file (substring file 0 -3))
   (load file))
 (when recursive
   (dolist (file (directory-files dir t))
     (if (and (file-directory-p file)
              (not (string-match "/\\.\\.?$" file)))
         (load-directory file recursive)))))

If I'm not mistaken that's about 90% of what you want, and makes using
your own load-dir at least as easy as using your own custom-file.

-PJ

Re: user-controlled load-path extension: load-dir

[Evans Winner]

,------ Ted Zlatanov wrote ------
|    Chong, Stefan, can I take this and the lack of new
|    posts in this thread to mean it's OK to commit an
|    experimental implementation of the load-dir idea?  I
|    don't know if Ben Key's paperwork is in progress but as
|    you observed it's not terribly complicated code so I
|    can implement it in a functionally similar way quickly
|    (adding his work once we have his copyright
|    assignment).

Is it normal to scoop people like this?  It seems like a
good incentive for people not to bother trying to contribute
to a project.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Wed, 16 Mar 2011 14:11:59 -0600 Evans Winner <ego111@gmail.com> wrote: 

EW> ,------ Ted Zlatanov wrote ------
EW> |    Chong, Stefan, can I take this and the lack of new
EW> |    posts in this thread to mean it's OK to commit an
EW> |    experimental implementation of the load-dir idea?  I
EW> |    don't know if Ben Key's paperwork is in progress but as
EW> |    you observed it's not terribly complicated code so I
EW> |    can implement it in a functionally similar way quickly
EW> |    (adding his work once we have his copyright
EW> |    assignment).

EW> Is it normal to scoop people like this?  It seems like a
EW> good incentive for people not to bother trying to contribute
EW> to a project.

I rewrote the code entirely and credited you and Ben in the headers.
The code is pretty trivial, especially after I cut out the
byte-compilation and the error trapping.  I think calling it "scooping"
is a big stretch since I proposed the idea too.

Ben's original code had naming and other issues which I didn't see
corrected.  Once it's corrected we can discuss what parts of it we need
incorporated.  But he hasn't said that he's willing to do the assignment
paperwork... and I was impatient enough to do the rewrite.

Ted

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> (dolist (f (directory-files dir t "\\.el$"))
                                         ^^^
                                         \\'

-- Stefan

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

Hello,

For what it is worth I signed the necessary copyright assignment papers
years ago.  I have been a member of the Emacs project since Wednesday,
November 13, 2002.  If you go to the Emacs Project Memberlist page you will
see me listed as Ben Key <bkey1>.  I simply have not been an active
contributor to Emacs for the past several years.

I also have been very busy at work for the last several weeks and have not
had a chance yet to make the various suggested name changes to the functions
and variables.  In fact I am on a business trip now, the second in 3 weeks.

I do not mind what Ted has done though.

I do think it is important to trap errors though.  If you do not, and you
call this new load directory function in your .emacs file, Emacs
initialization will abort at the first error leaving your Emacs session half
configured (any code positioned after the call to load directory will not
run).  I do not think this is a good idea.  I do not feel very strongly
about this though.

[-- Attachment #2: Type: text/html, Size: 1069 bytes --]

Re: user-controlled load-path extension: load-dir

[Evans Winner]

,------ Ben Key wrote ------
|    I do think it is important to trap errors though.  If
|    you do not, and you call this new load directory
|    function in your .emacs file, Emacs initialization will
|    abort at the first error leaving your Emacs session
|    half configured (any code positioned after the call to
|    load directory will not run).  I do not think this is a
|    good idea.  I do not feel very strongly about this
|    though.

I am not sure if you are talking about having Emacs silently
ignore files in the loaded directory which don't load
properly.  Personally, I would prefer such a function do no
error handling at all with the exception of perhaps making
sure its input argument is indeed a directory name.  When a
library is loaded by this or any other function on startup I
want Emacs to stop there and let me know about it, which is
the normal behavior at them moment I believe.

By the way, a version of `load-dir' that I sent Ben Key a
week or so ago mostly/partly includes handling of gzipped
libraries (or whatever is returned by `get-load-suffixes')
which seemed a useful variation, though it occurs to me that
I don't know if that is portable across platforms.

For, as usual, what it may be worth.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 3720 bytes --]

On Thu, 17 Mar 2011 00:22:10 -0400 Ben Key <bkey76@gmail.com> wrote: 

BK> For what it is worth I signed the necessary copyright assignment papers
BK> years ago.  I have been a member of the Emacs project since Wednesday,
BK> November 13, 2002.  If you go to the Emacs Project Memberlist page you will
BK> see me listed as Ben Key <bkey1>.  I simply have not been an active
BK> contributor to Emacs for the past several years.

Sorry Ben, I just checked the ChangeLogs.  I put you in the authors list
of user-load-dirs.el, which is attached below.  Please suggest any
changes you think are necessary.

BK> I do think it is important to trap errors though.  If you do not, and you
BK> call this new load directory function in your .emacs file, Emacs
BK> initialization will abort at the first error leaving your Emacs session half
BK> configured (any code positioned after the call to load directory will not
BK> run).  I do not think this is a good idea.  I do not feel very strongly
BK> about this though.

I think it's a valid way to do it, but the way I think of it (as a
logical extension to .emacs) errors should not be trapped.

I made it an option.

On Thu, 17 Mar 2011 00:31:34 -0600 Evans Winner <ego111@gmail.com> wrote: 

EW> By the way, a version of `load-dir' that I sent Ben Key a
EW> week or so ago mostly/partly includes handling of gzipped
EW> libraries (or whatever is returned by `get-load-suffixes')
EW> which seemed a useful variation, though it occurs to me that
EW> I don't know if that is portable across platforms.

It seems to work OK and automatically filters out "." and ".." so it's a
good idea.  I added that to user-load-dirs.el as well.

On Wed, 16 Mar 2011 21:42:17 -0700 PJ Weisberg <pj@irregularexpressions.net> wrote: 

PW> I only saw Stefan say to go ahead with a load-directory function,

I know, but a package IMO will provide a better user experience because
of the number of possible options.  We already have recursive and
ignore-errors; I imagine the list will only grow.  Plus the loaded cache
in `user-load-dirs-loaded' is pretty crucial to avoid double-loading
files; remember Stephen Turnbull's concerns about installing hooks twice
for instance.  All of that is a pain to handle without a package.

PW> which I assumed would live in files.el next to load-file and look
PW> something like this:

PW> (defun load-directory (dir recursive)
PW>  "Load all Lisp files in DIR.  If RECURSIVE is non-nil,
PW> recursively call load-directory for all subdirectories of dir."
PW>  (interactive "DLoad Dir: \nP")
PW>  (dolist (file (directory-files dir t ".*\\.el$"))
PW>    ;; Cut off the .el so load will pick up the .elc version if it's there
PW>    (setq file (substring file 0 -3))
PW>    (load file))
PW>  (when recursive
PW>    (dolist (file (directory-files dir t))
PW>      (if (and (file-directory-p file)
PW>               (not (string-match "/\\.\\.?$" file)))
PW>          (load-directory file recursive)))))

PW> If I'm not mistaken that's about 90% of what you want, and makes using
PW> your own load-dir at least as easy as using your own custom-file.

I will use `file-name-sans-extension' instead of the substring to handle
all the possible values from `get-load-suffixes', but yeah I think using
the bare file name is better than explicitly loading it with the
extension.  Thanks for that (and I think both Ben and Evans had
something similar).

If `user-load-dirs-load' is autoloaded then it's just as easy to manage
at the user level as the `load-directory' function you suggest.

The library can be made into an ELPA package easily and thus used in
older Emacsen.  My goal, though, is to call `user-load-dirs-load' in the
Emacs startup, as I've mentioned.

Ted


[-- Attachment #2: user-load-dirs.el --]
[-- Type: application/emacs-lisp, Size: 3830 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

PW> I only saw Stefan say to go ahead with a load-directory function,

Indeed.

> I know, but a package IMO will provide a better user experience because
> of the number of possible options.  We already have recursive and
> ignore-errors; I imagine the list will only grow.  Plus the loaded cache
> in `user-load-dirs-loaded' is pretty crucial to avoid double-loading
> files; remember Stephen Turnbull's concerns about installing hooks twice
> for instance.  All of that is a pain to handle without a package.

Sounds fine, but then make it an ELPA package.  If/when experience shows
it's a package many users want/need, we can move it into core
(i.e. bundle it with the standard Emacs distribution).


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

On Thu, 17 Mar 2011 16:11:47 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
SM> Ted wrote:
>> I know, but a package IMO will provide a better user experience because
>> of the number of possible options.  We already have recursive and
>> ignore-errors; I imagine the list will only grow.  Plus the loaded cache
>> in `user-load-dirs-loaded' is pretty crucial to avoid double-loading
>> files; remember Stephen Turnbull's concerns about installing hooks twice
>> for instance.  All of that is a pain to handle without a package.

SM> Sounds fine, but then make it an ELPA package.  If/when experience shows
SM> it's a package many users want/need, we can move it into core
SM> (i.e. bundle it with the standard Emacs distribution).

OK. I've attached the library (now called load-dir.el, which I think is
much better than user-load-*, especially since the function names are
shorter).  If you find it acceptable I'll commit it to the ELPA branch
and deploy it (or let me know what more is needed).  I modified the
comments and license a bit as well.  It should be possible to include it
like rainbow-mode, directly as a single file.

The Commentary was adjusted to reflect that the user needs to explicitly
call (load-dirs) in their .emacs.

Thanks
Ted


[-- Attachment #2: load-dir.el --]
[-- Type: application/emacs-lisp, Size: 3985 bytes --]

Re: user-controlled load-path extension: load-dir

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> ;; (setq load-dirs '("~/mystuff")) ;; or Customize it
Ted> ;; (require 'load-dir)
Ted> ;; (load-dirs)

Why not put an autoload cookie on load-dirs?
That would be friendlier for users.

Tom

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 17 Mar 2011 15:05:57 -0600 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
Ted> ;; (setq load-dirs '("~/mystuff")) ;; or Customize it
Ted> ;; (require 'load-dir)
Ted> ;; (load-dirs)

Tom> Why not put an autoload cookie on load-dirs?
Tom> That would be friendlier for users.

I didn't think of it, thanks.  I added those to my copy.

Ted

Re: user-controlled load-path extension: load-dir

[Glenn Morris]

Ted Zlatanov wrote:

> Tom> Why not put an autoload cookie on load-dirs?
> Tom> That would be friendlier for users.
>
> I didn't think of it, thanks.  I added those to my copy.

Noooo.... Please don't autoload defcustoms unless they really, really
need it.

Can't find the best reference, but see eg

http://lists.gnu.org/archive/html/emacs-devel/2010-08/msg01355.html

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On Thu, 17 Mar 2011 18:50:42 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> Ted Zlatanov wrote:
Tom> Why not put an autoload cookie on load-dirs?
Tom> That would be friendlier for users.
>> 
>> I didn't think of it, thanks.  I added those to my copy.

GM> Noooo.... Please don't autoload defcustoms unless they really, really
GM> need it.

OK.  See attached.

On Thu, 17 Mar 2011 22:21:33 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> OK. I've attached the library (now called load-dir.el, which I think is
>> much better than user-load-*, especially since the function names are

SM> The mapcar call should be replaced by mapc or dolist.  As mentioned,
SM> you may want to add an autoload cookie on the [`load-dirs']
SM> function.

I did the mapc.  Not sure about the autoload cookie, see above.  So it's
off for now.

SM> Also you'll want to replace `ignore-errors' with
SM> `with-demoted-errors'.  More importantly, if it's an ELPA package,
SM> it can hook itself via autoload cookies so the user doesn't even
SM> need to call load-dirs and can just set load-dirs instead.  E.g.

SM>    ;;;###autoload
SM>    (add-hook 'after-init-hook 'load-dirs)

Done.  Thanks for the suggestions.  I wasn't sure if this is a good
default because of people who want to use this package but not on
startup.  But I guess that's a rare enough case that this is OK as the
default.

Ted


[-- Attachment #2: load-dir.el --]
[-- Type: application/emacs-lisp, Size: 4101 bytes --]

Re: user-controlled load-path extension: load-dir

[Dimitri Fontaine]

Ted Zlatanov <tzz@lifelogs.com> writes:
> SM>    ;;;###autoload
> SM>    (add-hook 'after-init-hook 'load-dirs)

That's very good news.

> (defcustom load-dirs nil
>   "List of directories to load."
>   :group 'load-dir
>   :type '(repeat directory))

Any reason why this would not default to ("~/.emacs.d/load.d")?  That
would allow easier usage from code.  The bootstrapping from nil is
uneasy here, or I'm lacking imagination :)

Regards,
-- 
dim

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Fri, 18 Mar 2011 11:56:10 +0100 Dimitri Fontaine <dim@tapoueh.org> wrote: 

>> (defcustom load-dirs nil
>> "List of directories to load."
>> :group 'load-dir
>> :type '(repeat directory))

DF> Any reason why this would not default to ("~/.emacs.d/load.d")?  That
DF> would allow easier usage from code.  The bootstrapping from nil is
DF> uneasy here, or I'm lacking imagination :)

I want this library to do absolutely nothing unless the user explicitly
indicates otherwise.  That way adding it to the Emacs core will not
break any backwards compatibility or have any unexpected effects.

Ted

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> OK. I've attached the library (now called load-dir.el, which I think is
> much better than user-load-*, especially since the function names are

The mapcar call should be replaced by mapc or dolist.
As mentioned, you may want to add an autoload cookie on the
`loadirs' function.  Also you'll want to replace `ignore-errors' with
`with-demoted-errors'.
More importantly, if it's an ELPA package, it can hook itself via
autoload cookies so the user doesn't even need to call load-dirs and can
just set load-dirs instead.  E.g.

   ;;;###autoload
   (add-hook 'after-init-hook 'load-dirs)


        Stefan

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 1564 bytes --]

Hello,

I have attached to this message an updated version of load-dir.el.  In this
version, I have made several basic improvements.

   1. Each of the defcustom items now has a require property.  The main
   benefit of this is that a user may now simply place load-dir.el in their
   load path, use M-x load-library to load 'load-dir' and then do a M-x
   customize-group to customize the 'load-dir' group, customize load-dirs, and
   restart Emacs.  There is no need to also manually edit their .emacs file to
   add a '(require 'load-dir)' in order to actually cause Emacs to load
   load-dir.el.
   2. Modified the load-dirs defcustom so that it can now accept several
   types via the "Value Menu" customize feature.
      1. nil, the default value, effectively causes load-dir.el to do
      nothing.
      2. t causes the default directory of ~/.emacs.d/load.d to be used
      (saves typing).
      3. A single directory causes files in that directory to be loaded.
      4. A list of directories causes all files in every directory in the
      list to be loaded.

Note that the error handling code does not appear to be working as
expected.  I set load-dir-ignore-errors to t.  I then placed a file error.el
in my load-dir directory that contains a deliberate error.  The expected
behavior is for Emacs initialization to complete with perhaps a message
telling me about the error.  Instead, Emacs initialization halts at the
error and the "*GNU Emacs*" buffer is never displayed.  I will look into
this latter.

I hope that you find this changes useful.

[-- Attachment #1.2: Type: text/html, Size: 1849 bytes --]

[-- Attachment #2: load-dir.el --]
[-- Type: application/octet-stream, Size: 4579 bytes --]

;;; load-dir.el --- load all Emacs Lisp files in given directories

;; Copyright (C) 2011 Free Software Foundation, Inc

;; Authors: Teodor Zlatanov <tzz@lifelogs.com>,
;;          Ben Key <bkey76@gmail.com>
;; With-Help-From: Evans Winner <ego111@gmail.com>, PJ Weisberg <pj@irregularexpressions.net>
;; Version: 0.0.1
;; Keywords: lisp, files, convenience

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; This library will load all Lisp files found in the `load-dirs' variable.
;; (you may also want to set `load-dir-debug', `load-dir-recursive',
;;  and `load-dir-ignore-errors')

;; Normal usage in .emacs:

;; (setq load-dirs '("~/mystuff")) ;; or Customize it
;; (require 'load-dir) ;; this will add `load-dirs' to your `after-init-hook'

;; Then after startup:

;; Explicitly load new files only...
;; M-x load-dirs

;; Or reload all your files...
;; M-x load-dirs-reload

;;; Code:

(eval-when-compile (require 'cl))

(defgroup load-dir nil
  "Automatically load all Emacs Lisp files in given directories."
  :group 'initialization)

(defcustom load-dir-debug t
  "Debugging messages toggle, default to t."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dir-recursive nil
  "List of directories to load."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dir-ignore-errors nil
  "Whether errors in the loaded files should be ignored."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dirs nil
  "This variable allows you to define which directories should be loaded in one
of several ways.

If load-dirs is nil, no directories are loaded.  This is the default behavior.
If load-dirs is t, only files in the default directory, which is ~/.emacs.d/load.d,
will be loaded.
If load-dirs is a single directory, only files in that directory will be loaded.
If load-dirs is a list of directories, all files found in all the directories in
the list will be loaded."
  :group 'load-dir
  :require 'load-dir
  :type '(choice (const t) (const nil) directory (repeat directory)))

(defun load-dirs ()
  "Load all Emacs Lisp files in `load-dirs'.
Will not load a file twice (use `load-dir-reload' for that).
Recurses into subdirectories if `load-dir-recursive' is t."
  (interactive)
  ;; avoid the case where users inadvertently set `load-dirs' to a string
  (mapc 'load-dir-one (if (eq load-dirs t)
						  (list (expand-file-name "~/.emacs.d/load.d"))
						(if (stringp load-dirs)
							(list load-dirs)
						  load-dirs))))

;;;###autoload
(defun load-dirs-reload ()
  "Load all Emacs Lisp files in `load-dirs'.
Clears the list of loaded files and just calls `load-dir-load'."
  (interactive)
  (setq load-dir-loaded nil)
  (load-dirs))

(defvar load-dir-loaded nil
  "List of already loaded files.")

(defun load-dir-one (dir)
  "Load all Emacs Lisp files in DIR.
Recurses into subdirectories if `load-dir-recursive' is t."
  (load-dir-debug "Loading Emacs Lisp code from %s" dir)
  (let ((suffixes (get-load-suffixes)))
    (dolist (f (and (file-exists-p dir)
                    (file-directory-p dir)
                    (directory-files dir t)))
      (when (and (not (file-directory-p f))
                 (member (file-name-extension f t) suffixes))
        (setq f (file-name-sans-extension f))
        (if (member f load-dir-loaded)
            (load-dir-debug "Skipping %s, it's already loaded." f)
          (if load-dir-ignore-errors
              (with-demoted-errors (load f))
            (load f))
          (add-to-list 'load-dir-loaded f))))

    (when load-dir-recursive
      (dolist (f (directory-files dir t))
        (when (file-directory-p f)
          (load-dir-one f))))))

(defun load-dir-debug (&rest args)
  "Print a debug message like `message' if `load-dir-debug' is set."
  (when load-dir-debug
    (apply 'message args)))

;;;###autoload
(add-hook 'after-init-hook 'load-dirs)

(provide 'load-dir)
;;; load-dir.el ends here

Re: user-controlled load-path extension: load-dir

[Juanma Barranquero]

On Sat, Mar 19, 2011 at 05:10, Ben Key <bkey76@gmail.com> wrote:

> I have attached to this message an updated version of load-dir.el.

(defcustom load-dir-recursive nil
  "List of directories to load."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

That docstring can't possibly be right :-)


    Juanma

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 380 bytes --]

On Sat, Mar 19, 2011 at 9:29 AM, Juanma Barranquero <lekktu@gmail.com>wrote:

>
> (defcustom load-dir-recursive nil
>  "List of directories to load."
>  :group 'load-dir
>  :require 'load-dir
>  :type 'boolean)
>
> That docstring can't possibly be right :-)
>
>
You are absolutely correct.  That was not one of the lines I changed
though.  The attached file fixes this docstring.

[-- Attachment #1.2: Type: text/html, Size: 883 bytes --]

[-- Attachment #2: load-dir.el --]
[-- Type: application/octet-stream, Size: 4635 bytes --]

;;; load-dir.el --- load all Emacs Lisp files in given directories

;; Copyright (C) 2011 Free Software Foundation, Inc

;; Authors: Teodor Zlatanov <tzz@lifelogs.com>,
;;          Ben Key <bkey76@gmail.com>
;; With-Help-From: Evans Winner <ego111@gmail.com>, PJ Weisberg <pj@irregularexpressions.net>
;; Version: 0.0.1
;; Keywords: lisp, files, convenience

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; This library will load all Lisp files found in the `load-dirs' variable.
;; (you may also want to set `load-dir-debug', `load-dir-recursive',
;;  and `load-dir-ignore-errors')

;; Normal usage in .emacs:

;; (setq load-dirs '("~/mystuff")) ;; or Customize it
;; (require 'load-dir) ;; this will add `load-dirs' to your `after-init-hook'

;; Then after startup:

;; Explicitly load new files only...
;; M-x load-dirs

;; Or reload all your files...
;; M-x load-dirs-reload

;;; Code:

(eval-when-compile (require 'cl))

(defgroup load-dir nil
  "Automatically load all Emacs Lisp files in given directories."
  :group 'initialization)

(defcustom load-dir-debug t
  "Debugging messages toggle, default to t."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dir-recursive nil
  "Whether to load files in each load-dir and all subdirectories or just each load-dir."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dir-ignore-errors nil
  "Whether errors in the loaded files should be ignored."
  :group 'load-dir
  :require 'load-dir
  :type 'boolean)

(defcustom load-dirs nil
  "This variable allows you to define which directories should be loaded in one
of several ways.

If load-dirs is nil, no directories are loaded.  This is the default behavior.
If load-dirs is t, only files in the default directory, which is ~/.emacs.d/load.d,
will be loaded.
If load-dirs is a single directory, only files in that directory will be loaded.
If load-dirs is a list of directories, all files found in all the directories in
the list will be loaded."
  :group 'load-dir
  :require 'load-dir
  :type '(choice (const t) (const nil) directory (repeat directory)))

(defun load-dirs ()
  "Load all Emacs Lisp files in `load-dirs'.
Will not load a file twice (use `load-dir-reload' for that).
Recurses into subdirectories if `load-dir-recursive' is t."
  (interactive)
  ;; avoid the case where users inadvertently set `load-dirs' to a string
  (mapc 'load-dir-one (if (eq load-dirs t)
						  (list (expand-file-name "~/.emacs.d/load.d"))
						(if (stringp load-dirs)
							(list load-dirs)
						  load-dirs))))

;;;###autoload
(defun load-dirs-reload ()
  "Load all Emacs Lisp files in `load-dirs'.
Clears the list of loaded files and just calls `load-dir-load'."
  (interactive)
  (setq load-dir-loaded nil)
  (load-dirs))

(defvar load-dir-loaded nil
  "List of already loaded files.")

(defun load-dir-one (dir)
  "Load all Emacs Lisp files in DIR.
Recurses into subdirectories if `load-dir-recursive' is t."
  (load-dir-debug "Loading Emacs Lisp code from %s" dir)
  (let ((suffixes (get-load-suffixes)))
    (dolist (f (and (file-exists-p dir)
                    (file-directory-p dir)
                    (directory-files dir t)))
      (when (and (not (file-directory-p f))
                 (member (file-name-extension f t) suffixes))
        (setq f (file-name-sans-extension f))
        (if (member f load-dir-loaded)
            (load-dir-debug "Skipping %s, it's already loaded." f)
          (if load-dir-ignore-errors
              (with-demoted-errors (load f))
            (load f))
          (add-to-list 'load-dir-loaded f))))

    (when load-dir-recursive
      (dolist (f (directory-files dir t))
        (when (file-directory-p f)
          (load-dir-one f))))))

(defun load-dir-debug (&rest args)
  "Print a debug message like `message' if `load-dir-debug' is set."
  (when load-dir-debug
    (apply 'message args)))

;;;###autoload
(add-hook 'after-init-hook 'load-dirs)

(provide 'load-dir)
;;; load-dir.el ends here

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

On Sat, 19 Mar 2011 00:10:26 -0400 Ben Key <bkey76@gmail.com> wrote: 

BK>    1. Each of the defcustom items now has a require property.  The main
BK>    benefit of this is that a user may now simply place load-dir.el in their
BK>    load path, use M-x load-library to load 'load-dir' and then do a M-x
BK>    customize-group to customize the 'load-dir' group, customize load-dirs, and
BK>    restart Emacs.  There is no need to also manually edit their .emacs file to
BK>    add a '(require 'load-dir)' in order to actually cause Emacs to load
BK>    load-dir.el.

Cool.

BK>    2. Modified the load-dirs defcustom so that it can now accept several
BK>    types via the "Value Menu" customize feature.
BK>       1. nil, the default value, effectively causes load-dir.el to do
BK>       nothing.
BK>       2. t causes the default directory of ~/.emacs.d/load.d to be used
BK>       (saves typing).
BK>       3. A single directory causes files in that directory to be loaded.
BK>       4. A list of directories causes all files in every directory in the
BK>       list to be loaded.

I added :tag properties too so it looks nice in Customize and changed
the double `if' you had to handle these load-dirs variations to a
`cond'.  But yeah, this is a good change.  I was thinking that maybe
~/.emacs.d/load.d should be customizable (it shows up in 3 places and
may need to vary on W32) but it works either way.

On Sat, 19 Mar 2011 14:29:30 +0100 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> (defcustom load-dir-recursive nil
JB>   "List of directories to load."

JB> That docstring can't possibly be right :-)

Thanks, fixed.  Version 0.0.2 attached for your review.

Ted


[-- Attachment #2: load-dir.el --]
[-- Type: application/emacs-lisp, Size: 4969 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>    1. Each of the defcustom items now has a require property.
>    The main benefit of this is that a user may now simply place
>    load-dir.el in their load path, use M-x load-library to load
>    'load-dir' and then do a M-x customize-group to customize the
>    'load-dir' group, customize load-dirs, and restart Emacs.  There is
>    no need to also manually edit their .emacs file to add a '(require
>    'load-dir)' in order to actually cause Emacs to load load-dir.el.

If it's meant as an ELPA package, this is unneeded, since just
installing the package can cause installation of some autoloads.
This is important because the use of :require in defcustoms is generally
discouraged (it places the feature name (i.e. the file name) into the
user's custom-file, so if the feature gets moved to another file or of
the file gets renamed, previous cutomizations tend to signals errors
about not finding the old file name).


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Sat, 19 Mar 2011 22:58:15 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> 1. Each of the defcustom items now has a require property.
>> The main benefit of this is that a user may now simply place
>> load-dir.el in their load path, use M-x load-library to load
>> 'load-dir' and then do a M-x customize-group to customize the
>> 'load-dir' group, customize load-dirs, and restart Emacs.  There is
>> no need to also manually edit their .emacs file to add a '(require
>> 'load-dir)' in order to actually cause Emacs to load load-dir.el.

SM> If it's meant as an ELPA package, this is unneeded, since just
SM> installing the package can cause installation of some autoloads.
SM> This is important because the use of :require in defcustoms is generally
SM> discouraged (it places the feature name (i.e. the file name) into the
SM> user's custom-file, so if the feature gets moved to another file or of
SM> the file gets renamed, previous cutomizations tend to signals errors
SM> about not finding the old file name).

OK, I've taken those requires out of my version.  Is it possible to put
it in the ELPA branch?

Ted

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Sun, 20 Mar 2011 07:05:14 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Sat, 19 Mar 2011 22:58:15 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>>> 1. Each of the defcustom items now has a require property.
>>> The main benefit of this is that a user may now simply place
>>> load-dir.el in their load path, use M-x load-library to load
>>> 'load-dir' and then do a M-x customize-group to customize the
>>> 'load-dir' group, customize load-dirs, and restart Emacs.  There is
>>> no need to also manually edit their .emacs file to add a '(require
>>> 'load-dir)' in order to actually cause Emacs to load load-dir.el.

SM> If it's meant as an ELPA package, this is unneeded, since just
SM> installing the package can cause installation of some autoloads.
SM> This is important because the use of :require in defcustoms is generally
SM> discouraged (it places the feature name (i.e. the file name) into the
SM> user's custom-file, so if the feature gets moved to another file or of
SM> the file gets renamed, previous cutomizations tend to signals errors
SM> about not finding the old file name).

TZ> OK, I've taken those requires out of my version.  Is it possible to put
TZ> it in the ELPA branch?

Do you want me to post the version with those changes or can I just
commit it to the ELPA branch?

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

> Do you want me to post the version with those changes or can I just
> commit it to the ELPA branch?

You can commit it into the ELPA (tho I think it'd make more sense to
merge yours and Ben's first).


        Stefan

Re: user-controlled load-path extension: load-dir

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

Stefan Monnier wrote:

> You can commit it into the ELPA (tho I think it'd make more sense to
> merge yours and Ben's first).

The latest version Ted released on Sat, 19 Mar 2011 already has all my
changes merged in.  Ted's implementation was actually cleaner than my
original implementation.  As far as I am concerned, he can simply use that
version with the :require tags removed from the defcustom's.

[-- Attachment #2: Type: text/html, Size: 476 bytes --]

Re: user-controlled load-path extension: load-dir

[Stefan Monnier]

>> You can commit it into the ELPA (tho I think it'd make more sense to
>> merge yours and Ben's first).

> The latest version Ted released on Sat, 19 Mar 2011 already has all my
> changes merged in.  Ted's implementation was actually cleaner than my
> original implementation.  As far as I am concerned, he can simply use that
> version with the :require tags removed from the defcustom's.

Great.  So, go ahead,


        Stefan

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Thu, 24 Mar 2011 10:38:08 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> You can commit it into the ELPA (tho I think it'd make more sense to
>>> merge yours and Ben's first).

>> The latest version Ted released on Sat, 19 Mar 2011 already has all my
>> changes merged in.  Ted's implementation was actually cleaner than my
>> original implementation.  As far as I am concerned, he can simply use that
>> version with the :require tags removed from the defcustom's.

SM> Great.  So, go ahead,

I committed it (lucky rev 13) but haven't deployed on elpa.gnu.org.  If
you or Chong could do a final check on the elpa branch that would be
great.

Thanks
Ted

Re: user-controlled load-path extension: load-dir

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 24 Mar 2011 10:38:08 -0400 Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>
>>>> You can commit it into the ELPA (tho I think it'd make more sense to
>>>> merge yours and Ben's first).
>
>>> The latest version Ted released on Sat, 19 Mar 2011 already has all my
>>> changes merged in.  Ted's implementation was actually cleaner than my
>>> original implementation.  As far as I am concerned, he can simply use that
>>> version with the :require tags removed from the defcustom's.
>
> SM> Great.  So, go ahead,
>
> I committed it (lucky rev 13) but haven't deployed on elpa.gnu.org.  If
> you or Chong could do a final check on the elpa branch that would be
> great.

Looks mostly OK, but instead of referring to .emacs.d, use the variable
`user-emacs-directory' or the function `locate-user-emacs-file'.

Also, the current description, "load all Emacs Lisp files in given
directories", is too vague; if you could improve it, that would be good.
It should be stated more clearly, there and in the commentary, that
installing this package alters Emacs startup, by updating
after-init-hook to load all files in the specified directory.

Re: user-controlled load-path extension: load-dir

[Ted Zlatanov]

On Sat, 02 Apr 2011 14:35:43 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Looks mostly OK, but instead of referring to .emacs.d, use the variable
CY> `user-emacs-directory' or the function `locate-user-emacs-file'.

CY> Also, the current description, "load all Emacs Lisp files in given
CY> directories", is too vague; if you could improve it, that would be good.
CY> It should be stated more clearly, there and in the commentary, that
CY> installing this package alters Emacs startup, by updating
CY> after-init-hook to load all files in the specified directory.

I did both of these, committed to the ELPA branch.

Thanks
Ted