emacs trying to load too many files at startup

emacs trying to load too many files at startup

[Dan Nicolaescu]

On an x86 Fedora Core 4 machine I run this:

strace -o /tmp/emacs.strace ./emacs -q --no-sile-file

The /tmp/emacs.strace file shows that emacs is trying to load at
startup a lot of files that are dumped anyway so there should be no
reason to load them at startups. 

This is seen by looking at stat system calls like:
stat64("/home/dann/packages/emacs-CVS/share/emacs/22.0.50/site-lisp/replace.elc", 0xbf8c767c)

Other files loaded: abbrev, textmodes/fill, emacs-lisp/lisp-mode, etc, etc

There are about 2400 stat syscalls during the startup. 
emacs-21.4 does about 230 stat64 syscalls... 

When running from the local disk this is unlikely to cause a problem,
but it might be a problem when using a slow NFS server...

Does anyone know what is going on? 

Thanks

                --dan

Re: emacs trying to load too many files at startup

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sat, 19 Nov 2005 10:31:16 -0800
> 
> The /tmp/emacs.strace file shows that emacs is trying to load at
> startup a lot of files that are dumped anyway so there should be no
> reason to load them at startups. 
> 
> This is seen by looking at stat system calls like:
> stat64("/home/dann/packages/emacs-CVS/share/emacs/22.0.50/site-lisp/replace.elc", 0xbf8c767c)
> 
> Other files loaded: abbrev, textmodes/fill, emacs-lisp/lisp-mode, etc, etc
> 
> There are about 2400 stat syscalls during the startup. 
> emacs-21.4 does about 230 stat64 syscalls... 
> 
> When running from the local disk this is unlikely to cause a problem,
> but it might be a problem when using a slow NFS server...
> 
> Does anyone know what is going on? 

The easiest and the most accurate way of answering that is to put a
breakpoint in stat64, and then do a bt each time the breakpoint
triggers.

Re: emacs trying to load too many files at startup

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > From: Dan Nicolaescu <dann@ics.uci.edu>
  > > Date: Sat, 19 Nov 2005 10:31:16 -0800
  > > 
  > > The /tmp/emacs.strace file shows that emacs is trying to load at
  > > startup a lot of files that are dumped anyway so there should be no
  > > reason to load them at startups. 
  > > 
  > > This is seen by looking at stat system calls like:
  > > stat64("/home/dann/packages/emacs-CVS/share/emacs/22.0.50/site-lisp/replace.elc", 0xbf8c767c)
  > > 
  > > Other files loaded: abbrev, textmodes/fill, emacs-lisp/lisp-mode, etc, etc
  > > 
  > > There are about 2400 stat syscalls during the startup. 
  > > emacs-21.4 does about 230 stat64 syscalls... 
  > > 
  > > When running from the local disk this is unlikely to cause a problem,
  > > but it might be a problem when using a slow NFS server...
  > > 
  > > Does anyone know what is going on? 
  > 
  > The easiest and the most accurate way of answering that is to put a
  > breakpoint in stat64, and then do a bt each time the breakpoint
  > triggers.

stat64 is a syscall, so you cannot set a breakpoint on it. I set a
breakpoint on Fexpand_file_name instead and that lead to:

(gdb)  xbacktrace
"locate-file-internal"
"locate-file"
"mapcar"
"command-line"
"normal-top-level"


And that seems to indicate this code:

  ;; Convert preloaded file names to absolute.
  (setq load-history
	(mapcar (lambda (elt)
		  (if (and (stringp (car elt))
			   (not (file-name-absolute-p (car elt))))
		      (cons (locate-file (car elt) load-path
					 load-suffixes)
			    (cdr elt))
		    elt))
		load-history))

This is the change that introduced the above code:

revision 1.381
date: 2005/10/21 17:20:45;  author: rms;  state: Exp;  lines: +11 -0
(command-line): Convert library names in `load-history' to absolute file names.

The question is: should anything be done about this? 
    
        --dan

Re: emacs trying to load too many files at startup

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sat, 19 Nov 2005 11:49:57 -0800
> 
>   > The easiest and the most accurate way of answering that is to put a
>   > breakpoint in stat64, and then do a bt each time the breakpoint
>   > triggers.
> 
> stat64 is a syscall, so you cannot set a breakpoint on it.

Of course you can put a breakpoint on stat64, since it's a normal
function that issues a syscall, not the syscall itself.  If you tried
and failed, it's probably because glibc plays all kinds of redirection
and weak-alias games.  Try "rbreak stat64", and you will see the name
of the _real_ function that issues the syscall.  But that's an aside,
it sounds like you found the culprit.

> I set a breakpoint on Fexpand_file_name instead and that lead to:
> 
> (gdb)  xbacktrace
> "locate-file-internal"
> "locate-file"
> "mapcar"
> "command-line"
> "normal-top-level"
> 
> 
> And that seems to indicate this code:
> 
>   ;; Convert preloaded file names to absolute.
>   (setq load-history
> 	(mapcar (lambda (elt)
> 		  (if (and (stringp (car elt))
> 			   (not (file-name-absolute-p (car elt))))
> 		      (cons (locate-file (car elt) load-path
> 					 load-suffixes)
> 			    (cdr elt))
> 		    elt))
> 		load-history))

Do we know that this code alone is responsible for those 2000 extra
calls to stat64?  Is it possible that there's some other code besides
Fexpand_file_name that causes some of those calls?

> This is the change that introduced the above code:
> 
> revision 1.381
> date: 2005/10/21 17:20:45;  author: rms;  state: Exp;  lines: +11 -0
> (command-line): Convert library names in `load-history' to absolute file names.
> 
> The question is: should anything be done about this? 

I don't know.  Anyone?

I'm guessing that this is converting non-absolute file names in
loaddefs.el into absolute file names.

Re: emacs trying to load too many files at startup

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > Cc: emacs-devel@gnu.org
  > > From: Dan Nicolaescu <dann@ics.uci.edu>
  > > I set a breakpoint on Fexpand_file_name instead and that lead to:
  > > 
  > > (gdb)  xbacktrace
  > > "locate-file-internal"
  > > "locate-file"
  > > "mapcar"
  > > "command-line"
  > > "normal-top-level"
  > > 
  > > 
  > > And that seems to indicate this code:
  > > 
  > >   ;; Convert preloaded file names to absolute.
  > >   (setq load-history
  > > 	(mapcar (lambda (elt)
  > > 		  (if (and (stringp (car elt))
  > > 			   (not (file-name-absolute-p (car elt))))
  > > 		      (cons (locate-file (car elt) load-path
  > > 					 load-suffixes)
  > > 			    (cdr elt))
  > > 		    elt))
  > > 		load-history))
  > 
  > Do we know that this code alone is responsible for those 2000 extra
  > calls to stat64?  Is it possible that there's some other code besides
  > Fexpand_file_name that causes some of those calls?

I'm reasonable sure. 4 versions of each file are stat-ed: .el .elc
.el.gz .elc.gz and each is looked up in a few directories in the
load-path... 

Re: emacs trying to load too many files at startup

[Richard M. Stallman]

The reason I designed this code this way

   ;; Convert preloaded file names to absolute.
   (setq load-history
 	(mapcar (lambda (elt)
 		  (if (and (stringp (car elt))
 			   (not (file-name-absolute-p (car elt))))
 		      (cons (locate-file (car elt) load-path
 					 load-suffixes)
 			    (cdr elt))
 		    elt))
 		load-history))

rather than recording the absolute file names right away is because
the files usually move between building and execution.  (For instance,
`make install' moves them.)  But there are surely faster ways to
do this job.  How about this?

*** startup.el	19 Nov 2005 18:34:55 -0500	1.389
--- startup.el	20 Nov 2005 13:24:18 -0500	
***************
*** 642,656 ****
    (set-locale-environment nil)
  
    ;; Convert preloaded file names to absolute.
!   (setq load-history
! 	(mapcar (lambda (elt)
! 		  (if (and (stringp (car elt))
! 			   (not (file-name-absolute-p (car elt))))
! 		      (cons (locate-file (car elt) load-path
! 					 load-suffixes)
! 			    (cdr elt))
! 		    elt))
! 		load-history))
  
    ;; Convert the arguments to Emacs internal representation.
    (let ((args (cdr command-line-args)))
--- 642,663 ----
    (set-locale-environment nil)
  
    ;; Convert preloaded file names to absolute.
!   (let ((lisp-dir
! 	 (file-name-directory
! 	  (locate-file "simple" load-path
! 		       load-suffixes))))
! 
!     (setq load-history
! 	  (mapcar (lambda (elt)
! 		    (if (and (stringp (car elt))
! 			     (not (file-name-absolute-p (car elt))))
! 			(cons (concat lisp-dir
! 				      (car elt)
! 				      (if (string-match "[.]el$" (car elt))
! 					  "" ".elc"))
! 			      (cdr elt))
! 		      elt))
! 		  load-history)))
  
    ;; Convert the arguments to Emacs internal representation.
    (let ((args (cdr command-line-args)))

Re: emacs trying to load too many files at startup

[Luc Teirlinck]

Dan Nicolaescu wrote:

   The /tmp/emacs.strace file shows that emacs is trying to load at
   startup a lot of files that are dumped anyway so there should be no
   reason to load them at startups. 

Do you really mean that Emacs is trying to load (execute the Lisp
expressions in) the file or that it is just trying to open those files?

   This is seen by looking at stat system calls like:

How can you use stat to check that Emacs executed the Lisp expressions
in a file?
   
   stat64 is a syscall, so you cannot set a breakpoint on it. I set a
   breakpoint on Fexpand_file_name instead and that lead to:

   (gdb)  xbacktrace
   "locate-file-internal"
   "locate-file"
   "mapcar"
   "command-line"
   "normal-top-level"

Unless I misunderstand something, the code in locate-file-internal
tries to open files, not load them:

{
  Lisp_Object file;
  int fd = openp (path, filename, suffixes, &file, predicate);
  if (NILP (predicate) && fd > 0)
    close (fd);
  return file;
}

Sincerely,

Luc.

Re: emacs trying to load too many files at startup

[Dan Nicolaescu]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

  > Dan Nicolaescu wrote:
  > 
  >    The /tmp/emacs.strace file shows that emacs is trying to load at
  >    startup a lot of files that are dumped anyway so there should be no
  >    reason to load them at startups. 
  > 
  > Do you really mean that Emacs is trying to load (execute the Lisp
  > expressions in) the file or that it is just trying to open those files?

At first I thought it is trying to load the files, but it seems that
it is just trying to stat the files.

Re: emacs trying to load too many files at startup

[Stefan Monnier]

> There are about 2400 stat syscalls during the startup. 
> emacs-21.4 does about 230 stat64 syscalls... 

We should be able to bring those way down:

1 - just before dumping, turn things like "lisp-mode.elc"
    into "emacs-lisp/lisp-mode.elc".  This requires many stat calls but
    who cares.
2 - at startup blindly prepend /foo/bar/22.0.50/lisp/ to each and every
    non-absolute item in load-history.  No stat calls here.


        Stefan

Re: emacs trying to load too many files at startup

[Richard M. Stallman]

    1 - just before dumping, turn things like "lisp-mode.elc"
	into "emacs-lisp/lisp-mode.elc".  This requires many stat calls but
	who cares.

My solution guesses at this at startup, but I would not mind
using your solution instead, if it seems to be better.