An alternative to a monolithic ~/.emacs init file

An alternative to a monolithic ~/.emacs init file

[Sebastian Tennant]

My ~/.emacs reads like this:

 ;;; generate auto-autoloads.el in ~/elisp/dotemacs/
 (let ((generated-autoload-file "~/elisp/dotemacs/auto-autoloads.el")
       (backup-inhibited t))
   (apply 'update-directory-autoloads '("~/elisp/lib/"))
   (kill-buffer (file-name-nondirectory generated-autoload-file)))

 ;;; load dotemacs/*.el  (including auto-autoloads.el)
 (mapc (lambda (f) (load f))
   (split-string
     (shell-command-to-string "find ~/elisp/dotemacs -name *.el")))

That's it, essentially.

With this arrangement, library functions I've written that I don't
always want to load at startup, but I do want to have to hand, go in
~/elisp/lib, and customisations (and functions I do always want to load
at startup) go in ~/elisp/dotemacs.

I can put my customisations and functions in as many .el files as I like
within their respective directories, making it much easier to organise
things generally.

And to ensure a library function is always to hand (but not loaded) I
simply need to remember to include an autoload cookie (;;;###autoload)
in the library file, and an autoload form will be dynamically
constructed and called at startup, i.e., there's no longer any need to
write autoload forms manually any more.

It's a win-win is it not?  It works for me.

Sebastian

Re: An alternative to a monolithic ~/.emacs init file

[Sebastian Tennant]

Quoth Sebastian Tennant <sebyte@smolny.plus.com>:
>  ;;; generate auto-autoloads.el in ~/elisp/dotemacs/
>  (let ((generated-autoload-file "~/elisp/dotemacs/auto-autoloads.el")
>        (backup-inhibited t))
>    (apply 'update-directory-autoloads '("~/elisp/lib/"))
>    (kill-buffer (file-name-nondirectory generated-autoload-file)))

No need to write:

  (apply 'update-directory-autoloads '("~/elisp/lib"))

(I originally wanted to call update-directory-autoloads on a list of
 directories, dynamically generated at startup.)

A simple:

  (update-directory-autoloads "~/elisp/lib") 

will suffice.

Sebastian

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

Sebastian Tennant wrote:
> My ~/.emacs reads like this:
>
>  ;;; generate auto-autoloads.el in ~/elisp/dotemacs/
>  (let ((generated-autoload-file "~/elisp/dotemacs/auto-autoloads.el")
>        (backup-inhibited t))
>    (apply 'update-directory-autoloads '("~/elisp/lib/"))
>    (kill-buffer (file-name-nondirectory generated-autoload-file)))
>
>  ;;; load dotemacs/*.el  (including auto-autoloads.el)
>  (mapc (lambda (f) (load f))
>    (split-string
>      (shell-command-to-string "find ~/elisp/dotemacs -name *.el")))
>
> That's it, essentially.
>
> With this arrangement, library functions I've written that I don't
> always want to load at startup, but I do want to have to hand, go in
> ~/elisp/lib, and customisations (and functions I do always want to load
> at startup) go in ~/elisp/dotemacs.
>
> I can put my customisations and functions in as many .el files as I like
> within their respective directories, making it much easier to organise
> things generally.
>
> And to ensure a library function is always to hand (but not loaded) I
> simply need to remember to include an autoload cookie (;;;###autoload)
> in the library file, and an autoload form will be dynamically
> constructed and called at startup, i.e., there's no longer any need to
> write autoload forms manually any more.
>
> It's a win-win is it not?  It works for me.
>
> Sebastian

Could you please elaborate on how this works a little?
I did try to understand this by following the info/help:
update-directory-autoloads -> update-file-autoloads -> loaddefs.el ->
magic cookies
And by now I am quite lost.
I really would like to have a tiny little .emacs file and a bunch of
files and directories containing unrelated stuff

Re: An alternative to a monolithic ~/.emacs init file

[Sebastian Tennant]

Quoth rustom <rustompmody@gmail.com>:
> Could you please elaborate on how this works a little?  [...] I really
> would like to have a tiny little .emacs file and a bunch of files and
> directories containing unrelated stuff

If you don't care about autoloads (or you're not sure what they are) and
you just want a small ~/.emacs, then all you need is this:

  ;;; load dotemacs/*.el
  (mapc (lambda (f) (load f))
    (split-string
      (shell-command-to-string "find ~/elisp/dotemacs -name *.el")))

This routine simply loads each file ending '.el' found under the
directory ~/elisp/dotemacs.

Sebastian

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

On Nov 8, 6:08 pm, Sebastian Tennant <seb...@smolny.plus.com> wrote:
> Quoth rustom <rustompm...@gmail.com>:
>
> > Could you please elaborate on how this works a little?  [...] I really
> > would like to have a tiny little .emacs file and a bunch of files and
> > directories containing unrelated stuff
>
> If you don't care about autoloads (or you're not sure what they are) and
> you just want a small ~/.emacs, then all you need is this:
>
>   ;;; load dotemacs/*.el
>   (mapc (lambda (f) (load f))
>     (split-string
>       (shell-command-to-string "find ~/elisp/dotemacs -name *.el")))
>
> This routine simply loads each file ending '.el' found under the
> directory ~/elisp/dotemacs.
>
> Sebastian

Thanks. I think I understand autoload. Also I dont want needless
loading at emacs-startup -- its unlikely that in a given session I
will do C and python and ruby and org and tramp and elisp and .....
though I do use all these once in a while. What I dont understand are
magic cookies and how emacs uses them; or more correctly *When* emacs
uses them.  The elisp manual says:

>.These comments do nothing on their own, but they serve as a guide for the command `update-file-autoloads',
> which constructs calls to `autoload' and arranges to execute them when Emacs is built.

Does this mean I have to rebuild emacs if I want to use this ?!

Also what I guess is a related question -- What/Where/How is
loaddefs.el? It does not seem to be a fixed single file -- value of
the variable 'generated-autoloaded-file'  because many a package p
comes with its own p-loaddefs.el.  But then if loaddefs.el is not a
single file but a general class of files we are back to the same
problem that autoload sets out to solve namely:
 -- if emacs does not know about feature x it does not know what to do
 -- if emacs knows about x it does not need to have anything to do

In short I can (I think!) understand elisp code (been writing some for
15 odd years)
What I am unable to understand is what happens at what binding time:
1. emacs development time
2. emacs (re)build time
3. emacs start time
4. feature (first) reference time

Re: An alternative to a monolithic ~/.emacs init file

[Sebastian Tennant]

Quoth rustom <rustompmody@gmail.com>:
> Thanks. I think I understand autoload. Also I dont want needless
> loading at emacs-startup -- its unlikely that in a given session I
> will do C and python and ruby and org and tramp and elisp and .....
> though I do use all these once in a while. What I dont understand are
> magic cookies and how emacs uses them; or more correctly *When* emacs
> uses them.  The elisp manual says:
>
> "These comments do nothing on their own, but they serve as a guide for
>  the command `update-file-autoloads', which constructs calls to
>  'autoload' and arranges to execute them when Emacs is built."

> Does this mean I have to rebuild emacs if I want to use this ?!

First of all, autoloads exist precisely for the reason you've stated;
avoiding needless loading of functions at startup.  Consider the
function 'foo' defined in bar.el.  If your ~/.emacs includes the line

  (autoload 'foo "bar.el" "My function foo" t)

then Emacs knows where to find the function definition, if and when you
type the command 'M-x foo'.  The function itself is not loaded until you
call it, (the autoload call means Emacs is simply aware of the name of
the function, and the file where it is found), thus saving memory
resources and reducing startup time.  A better name than 'autoload'
might be:

 'load-later'

or,

 'dont-load-until-needed-but-heres-where-to-find-it-anyway'

I agree the explanation in the manual is unclear/inadequate.  You
certainly don't have to rebuild Emacs!  update-file-autoloads is called
when emacs is rebuilt, yes, but there's nothing to stop you calling
update-file-autoloads yourself.  It works by parsing a file and
constructing autoload calls (like the one above) based on occurrences of
magic cookies (;;;###autoload) in elisp code.  The constructed autoload
calls are then written to a file named according to the value of the
variable generated-autoload-file.

update-directory-autoloads works in exactly the same way, but on all the
files within a directory.  So,

 (let ((generated-autoload-file "~/elisp/dotemacs/auto-autoloads.el"))
   (update-directory-autoloads "~/elisp/lib/"))

works by temporarily binding the value of generated-autoload-file so
that autoload calls based on magic cookies found within files in
~/elisp/lib are written to a file within ~/elisp/dotemacs.

As for loaddefs.el, this is the equivalent of auto-autoloads.el above.
It is a file full of autoload calls created by the update-file-autoloads
function.  There are no less than six loaddefs.el on my machine but the
main one (/usr/share/emacs/22.1/lisp/loaddefs.el) is some 33,000 lines
long and I have no doubt it was generated once at build time, but is
loaded every time at startup, making all those functions available to
me, but not actually loading any of them until they're needed.

Perhaps someone else could confirm that this is the case.

Sebastian

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

Thanks Sebastian for a detailed response. But I cant see how it will
work. Let me try and collect the pieces of the jigsaw as I understand
them.
Firstly the organization

On Nov 9, 7:18 pm, Sebastian Tennant <seb...@smolny.plus.com> wrote:

> With this arrangement, library functions I've written that I don't always want to load at startup, but I do want to have
> to hand, go in ~/elisp/lib, and customisations (and functions I do always want to load at startup) go in
> ~/elisp/dotemacs.

Second the scales:
Lets assume that elisp/lib contains thousands of lines of lisp
distilled into say about 10 autoloads and a few customizations. In
general the user is never expected to use the 10 autoloads together.

Now the call:  (apply 'update-directory-autoloads '("~/elisp/lib"))
or   (update-directory-autoloads "~/elisp/lib")
will traverse the whole lib file-set -- all the thousands of lines --
at startup losing the advantage of autoloading.

For using an elisp module let us say foo I guess we need the following
three phases:

Phase 1 (To be run only when foo is added to elisp/lib)  Run update-
file-autloads and generate the autoloads into loaddefs.el in elisp/lib/
foo

Phase 2. Add a set of user customizations (called say
customizations.el) to elisp/lib/foo [Also one-time like the above]

Phase 3. In .emacs -- run on emacs startup -- Have a function say load-
loaddefs-from-directory that goes through each subdirectory in lib,
finds all files called loaddefs.el and loads them.

This way the .emacs can be nothing more than the two lines

(load-loaddefs-from-directory "~/elisp/lib")
(load-customizations-from-directory "~/elisp/lib")

And there are no superfluous file traversals at emacs startup time

Knowing emacs and its community I guess this functionality is already
available!

Question is what is it called :-)

Re: An alternative to a monolithic ~/.emacs init file

[Tom Tromey]

>>>>> ">" == rustom  <rustompmody@gmail.com> writes:

>> This way the .emacs can be nothing more than the two lines
>> (load-loaddefs-from-directory "~/elisp/lib")
>> (load-customizations-from-directory "~/elisp/lib")
>> And there are no superfluous file traversals at emacs startup time

>> Knowing emacs and its community I guess this functionality is already
>> available!

>> Question is what is it called :-)

ELPA works this way, more or less.  It isn't as simple as "drop in
some elisp and work", since it provides some other features that
require a bit more metadata.  But, the basic idea is the same: extract
autoloads and byte-compile at package install time; and read the
minimal number of files and only evaluate autoloads at package
evaluation time.

For more info:

    http://tromey.com/elpa/

There are some other, more minimal, elisp install packages.  They are
listed on the wiki.

Tom

Re: An alternative to a monolithic ~/.emacs init file

[Francisco Miguel Colaço]

Thu, 08 Nov 2007 11:53:08 -0800, rustom wrote:

> On Nov 8, 6:08 pm, Sebastian Tennant <seb...@smolny.plus.com> wrote:
>> Quoth rustom <rustompm...@gmail.com>:
>>
>> > Could you please elaborate on how this works a little?  [...] I
>> > really would like to have a tiny little .emacs file and a bunch of
>> > files and directories containing unrelated stuff
>>
>> If you don't care about autoloads (or you're not sure what they are)
>> and you just want a small ~/.emacs, then all you need is this:
>>
>>   ;;; load dotemacs/*.el
>>   (mapc (lambda (f) (load f))
>>     (split-string
>>       (shell-command-to-string "find ~/elisp/dotemacs -name *.el")))
>>


I use the following .emacs:

(set-language-environment "UTF-8")
(setq user-full-name "ABCDE"
      user-initials "abc"
      user-mail-address "abc@cde.com"
      user-dir-library "~/.local/share/emacs"
      user-dir-configuration "~/.config/emacs"
      user-dir-cache "~/.cache/emacs"
)

(setq load-path (append load-path (expand-file-name user-dir-library)))

;;; Define some needed functions.
(defun user-conf-filename (filename)
  "Gets the path of a configuration file, whose name is given.

  FILENAME is path appended to 'user-dir-configuration'.
  "
  (concat user-dir-configuration "/" filename))

;;; Load the user configuration.
(load (user-conf-filename "conf-user.el"))

Well, this puts all files within ~/.config/emacs.  Even ~/.emacs is a 
symlink to ~/.config/emacs/dotemacs.  This way:

1. I can zip all my emacs configuration by zipping one directory.
2. all I have to do is put in conf-user.el expressions like:

  (load (user-conf-filename "conf-fonts.el"))

  Of couse, I also maintain custom.el in the ~/.config/emacs directory by 
issuing, in .config/emacs/conf-user.el:

  (setq custom-file (user-conf-filename "custom.el"))
  (load custom-file)

  and all the emacs-specific files like ~/.projects.ede are moved intyo 
the directory by issuing the correspondent customize commands or setting 
their names in .emacs

  Works like a charm.

    Francisco Colaço

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

After my last post I I looked at my current .emacs file and realised
its more difficult than I thought:

Ideally if the separate aspects of the setup I want were independent
they could be put into separate directories and the corresponding
loaddefs and customizations loaded in any (non-deterministic) order.

However there are dependencies: eg to use python (with refactoring)
one needs bicyclerepairman; to use brm one needs pymacs.

What is worse there are emacs bugs whose correction needs careful
sequencing:
For example I need to put
(provide 'sb-info)
just before the brm-init otherwise I get 'recursive require' errors.
What in the name of ... is sb-info -- thats another OT matter

Re: An alternative to a monolithic ~/.emacs init file

[Tim X]

rustom <rustompmody@gmail.com> writes:

> After my last post I I looked at my current .emacs file and realised
> its more difficult than I thought:
>
> Ideally if the separate aspects of the setup I want were independent
> they could be put into separate directories and the corresponding
> loaddefs and customizations loaded in any (non-deterministic) order.
>
> However there are dependencies: eg to use python (with refactoring)
> one needs bicyclerepairman; to use brm one needs pymacs.
>

The common way to get around this is to number your files -
e.g. 50my-lib.el 60my-other-lib.el etc. This forces files to be loaded in a
specific order. 


> What is worse there are emacs bugs whose correction needs careful
> sequencing:
> For example I need to put
> (provide 'sb-info)
> just before the brm-init otherwise I get 'recursive require' errors.
> What in the name of ... is sb-info -- thats another OT matter
>
I remember seeing this one when I was running CVS emacs prior to the
release of emacs 22. Its a bug from the interaction between the speedbar
installed as part of standard emacs and a free standing versions of
speedbar. I ran into this on a Debian box when I had emacs 21 and the
speedbar package installed and then installed the CVS version of emacs 22.

Note that your 'fix' i.e. putting a (provide 'sb.info) is only hiding the
problem and not actually fixing it. If your running emacs 22 and still have
a stand-alone version of speedbar installed, you could just remove the
stand-alone version.

Tim

-- 
tcross (at) rapttech dot com dot au

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

On Nov 10, 10:13 pm, Tom Tromey <t...@tromey.com> wrote:
>
> ELPA works this way, more or less.  It isn't as simple as "drop in some elisp and work"

Thanks Tom. I checked out elpa and also install-elisp.  Considering we
are living in the age of SOA -- think of the success of things like
apt in the linux world or eggs in python gems in ruby  -- I suppose
its the in thing to be able to directly install an elisp file from a
web archive.

What I am looking for is something different:
- I have a bunch of my own customizations and autoloads rather than
ready made packages
- I of course want to autoload n rather than load n (for large n)
- The customize generated stuff only makes it much worse and more
monolithic.
- The .emacs file tends to become a kitchen sink kind of mess.

All this is not merely a question of aesthetics -- I like to sell
emacs to people but if I show a typical .emacs people would just run
away.

Note Francisco's solution does not separate out the autoload time from
the load time -- the basic problem as I see it.

So I am looking at an approach to distributing this kitchen
sink .emacs across different 'drawers' , each drawer (folder)
containing

1. autoloads loaded at emacs start time
2. customizations 'hooked' to the feature. [Admission: I dont
understand customizations very well]
3. the feature itself (This is usually absent because its a standard
mode in a standard place like cc-mode or python-mode etc)

I could of course write this infrastructure myself but I would be very
surprised if it had not already been written many times over before.

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

> Note that your 'fix' i.e. putting a (provide 'sb.info) is only hiding the
> problem and not actually fixing it. If your running emacs 22 and still have
> a stand-alone version of speedbar installed, you could just remove the
> stand-alone version.
>
> Tim

Just tried this. Evidently ecb has a dependency to speedbar.  So
removing speedbar means removing ecb.
Anyhow went ahead with this assuming that in emacs 22 these things are
in the base install.
This removed cedet-contrib, cogre, ecb, ede, semantic and speedbar
Now ecb does not work. speedbar standalone works but that is really
ugly compared to ecb.

I am running emacs-snapshot-gtk under debian etch
M-x version gives GNU Emacs 22.1.50.1

PS. Sorry for the reversed posts. google groups put my earlier post
later by about a day

Re: An alternative to a monolithic ~/.emacs init file

[Tim X]

rustom <rustompmody@gmail.com> writes:

>> Note that your 'fix' i.e. putting a (provide 'sb.info) is only hiding the
>> problem and not actually fixing it. If your running emacs 22 and still have
>> a stand-alone version of speedbar installed, you could just remove the
>> stand-alone version.
>>
>> Tim
>
> Just tried this. Evidently ecb has a dependency to speedbar.  So
> removing speedbar means removing ecb.
> Anyhow went ahead with this assuming that in emacs 22 these things are
> in the base install.
> This removed cedet-contrib, cogre, ecb, ede, semantic and speedbar
> Now ecb does not work. speedbar standalone works but that is really
> ugly compared to ecb.
>
> I am running emacs-snapshot-gtk under debian etch
> M-x version gives GNU Emacs 22.1.50.1
>

Ah, yep, your running into one of the issues with using a distro packaging
system. Packages like ecb depend on a stand alone speedbar package (as do
the others you listed, but don't know about speedbar now being packaged in
emacs itself. I'm not sure if you could use the force option to apt to
install ecb or not. The other issue may also be that I think the bundled
speedbar version in emacs 22 is older than the version in the stand alone
package (at least, this was the case some time back before emacs 22 was
released.)

From memory, when I looked at this the problem was emacs getting confused
due to mixups in the emacs bundled speedbar .el files and the ones that are
part of the stand alone version. Due to changes in these files, a circular
dependency arises where one file 'requires' the second file and the second
file 'requires' the first one. 

I'm very surprised your (provide 'sb-info) works though. Essentially, by
putting a provide statement into your file tells it that the definitions
have been loaded and there is no need to load the file. I would have
thought this would break things as the file has not been loaded (unless its
somehow loaded somewhere else). 

I'd check the bug reports for ecb - its likely a bug has been reported and
there could be a better fix provided. Alternatively, it may be possible to
get around the issue by re-jigging load paths so that a more consistent
behavior occurs. 

On your original question regarding breaking up your .emacs. I've taken a
very simple approach, but it works really well for me. 

All my customizations apart from the auto-generated 'custom' sections are
in their own files (broken up into groups of functionality). At the end of
each file, I just put a (provides blah). All the files are in ~/.emacs.d
I add the directory to my load path in .emacs and then just have (require
'blah)  line for the things I want loaded. 

This means my .emacs consists of only a couple of add-to-list statements
that add to my load-path and lines of require statements followed by the
customize section. 

What I like about this approach is that if I want to stop loading some bit
of my extensions, I just comment out the appropriate require line. I find
this quick and simple. Its only downside is that I have to edit my .emacs
to stop one of my extensions from being loaded. 

Originally, I used a system that was very similar to the way Debian loads
its various setup files from /etc/emacs/site-start.d et. al. This wasn't
bad because I could change what was loaded by simply moving the file out of
my .emacs.d directory and didn't need to edit the .emacs file. However, I
ended up dropping this approach, but to be honest, can't remember why. 

There is also the initz emacs package. This package allows you to split
your emacs config over various files, can control what is loaded based on
the version of emacs you are running and had some other fairly advanced
features. However, it was a bit overly complex or over engineered for my
needs. Probably very good if your someone who uses emacs and xemacs at the
same time.

tim

-- 
tcross (at) rapttech dot com dot au

Re: An alternative to a monolithic ~/.emacs init file

[rustom]

Thanks Tim for the pointers.



On Nov 13, 2:26 pm, Tim X <t...@nospam.dev.null> wrote:
> Ah, yep, your running into one of the issues with using a distro packaging
> system. Packages like ecb depend on a stand alone speedbar package (as do
> the others you listed, but don't know about speedbar now being packaged in
> emacs itself. I'm not sure if you could use the force option to apt to
> install ecb or not. The other issue may also be that I think the bundled
> speedbar version in emacs 22 is older than the version in the stand alone
> package (at least, this was the case some time back before emacs 22 was
> released.)

This is clearly a bigger problem than people are accepting: viz that
there are large systems of packages which are tolerably consistent in
themselves but dont talk nice to each other -- think of
apt <-> emacs
apt <-> ruby/python
eclipse <-> python
apt <-> webmin

I believe that a decent response to this issue will need:
1. A carefully thought-out standardized data oriented way (not XML
please!) for managing package repos
2. An event driven engine maybe along the lines of upstart see
https://wiki.ubuntu.com/ReplacementInit
3. built on an SOA backbone.  Yeah I know that programmers are likely
to think of this as unsubstantiated hype but when we do apt-get
install foo we are using an SOA