files.el: Once again impossible to turn off dir-settings

files.el: Once again impossible to turn off dir-settings

[T.V. Raman]

I've been watching the thread about forcing users to always take
the performance hit of walking up the directory tree with some
dismay. I notice that after the latest round of changes, it has
once again become impossible to disable project settings ---
other than locally advising hack-dir-local-variables ---
something the extreme ened of power users can certainly do -- (I
already have).

However, the design of dir-locals as implemented in files.el is
going to affect lots of users negatively -- especially those
 accessing files on network file systems.

I would still request that we enable users to selectively turn
off this feature --- I edit lots of things with Emacs, most of
them (none at present) use project settings --- and I dont see
why users in my position who also happen to use network file
systems should be asked to take this performance hit. Notice also
that the only way to discover the cause of this problem is to
toggle-debug-on-error, wait for the slowness to bite -- hit
ctrl-g -- chase the culprit down --- after which if you really
know your Emacs, you can advice hack-dir-local-variables to be a
no-op.

--Raman

-- 
Best Regards,
--raman


Email:  raman@users.sf.net
WWW:    http://emacspeak.sf.net/raman/
AIM:    emacspeak       GTalk: tv.raman.tv@gmail.com
PGP:    http://emacspeak.sf.net/raman/raman-almaden.asc
Google: tv+raman
IRC:    irc://irc.freenode.net/#emacs

Re: files.el: Once again impossible to turn off dir-settings

[Stefan Monnier]

> I've been watching the thread about forcing users to always take
> the performance hit of walking up the directory tree with some
> dismay. I notice that after the latest round of changes, it has
> once again become impossible to disable project settings ---
> other than locally advising hack-dir-local-variables ---
> something the extreme ened of power users can certainly do -- (I
> already have).

I thought we had fixed this slowness.  I often access files over the
network and have not bumped into this slowness, so I'm pretty sure it
can be avoided without turning off the feature.  Especially since the
"walk up the dir" is not done only for dir-settings but also for
VC backends, so it's pretty heavily used (meaning not that it's
important, but that we should have seen this slowness more often if it
was "normal").

Can we go back to trying to actually fix the slowness rather than
workaround it?


        Stefan

Re: files.el: Once again impossible to turn off dir-settings

[Juri Linkov]

> I notice that after the latest round of changes, it has once again
> become impossible to disable project settings --- other than locally
> advising hack-dir-local-variables --- something the extreme ened of
> power users can certainly do -- (I already have).

It is possible to disable dir-local variables by setting
`enable-local-variables' to nil.  However, this will also disable
file-local variables.

Perhaps we should add a similar variable `enable-dir-local-variables'
that will disable only dir-local variables.  It could also have an option
`remote' to disable searching upwards in remote filesystems.  (This option
could behave like the variable `dir-locals-chase-remote' in the package
`dir-locals.el').

-- 
Juri Linkov
http://www.jurta.org/emacs/

files.el: Once again impossible to turn off dir-settings

[T.V. Raman]

Stephane -
The slowness --- at least as I observe at my end, has not been
fixed.
In the caase of vc-backends, I'm able to avoid global crawls
across NFS filesystems by setting
;;; vc speed up:

(setq vc-ignore-dir-regexp
"\\`\\([\\/][\\/]\\|/net/\\|/home/\\|/afs/\\)\\'")
Notice the addition of /home above  -- in my case /home is nfs
mounted.

I strongly urge you to reconsider forcing all users to  take the
hit of searching for project settings.

For now, I've defadviced hack-dir-local-variables like so:
(defadvice hack-dir-local-variables (around fix-slowness pre act comp)
  "Restore democracy, restore speed"
nil)

-- 
Best Regards,
--raman

Title:  Research Scientist
Email:  raman@google.com
WWW:    http://emacspeak.sf.net/raman/
Google: tv+raman
GTalk:  raman@google.com, tv.raman.tv@gmail.com
PGP:    http://emacspeak.sf.net/raman/raman-almaden.asc



On 11/25/08, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> I've been watching the thread about forcing users to always take
>> the performance hit of walking up the directory tree with some
>> dismay. I notice that after the latest round of changes, it has
>> once again become impossible to disable project settings ---
>> other than locally advising hack-dir-local-variables ---
>> something the extreme ened of power users can certainly do -- (I
>> already have).
>
> I thought we had fixed this slowness.  I often access files over the
> network and have not bumped into this slowness, so I'm pretty sure it
> can be avoided without turning off the feature.  Especially since the
> "walk up the dir" is not done only for dir-settings but also for
> VC backends, so it's pretty heavily used (meaning not that it's
> important, but that we should have seen this slowness more often if it
> was "normal").
>
> Can we go back to trying to actually fix the slowness rather than
> workaround it?
>
>
>         Stefan
>
>

files.el: Once again impossible to turn off dir-settings

[T.V. Raman]

Correct.

I want to disable dir-local variables but keep the goodness that
is file-local variables.

-- 
Best Regards,
--raman

Title:  Research Scientist
Email:  raman@google.com
WWW:    http://emacspeak.sf.net/raman/
Google: tv+raman
GTalk:  raman@google.com, tv.raman.tv@gmail.com
PGP:    http://emacspeak.sf.net/raman/raman-almaden.asc



On 11/25/08, Juri Linkov <juri@jurta.org> wrote:
>> I notice that after the latest round of changes, it has once again
>> become impossible to disable project settings --- other than locally
>> advising hack-dir-local-variables --- something the extreme ened of
>> power users can certainly do -- (I already have).
>
> It is possible to disable dir-local variables by setting
> `enable-local-variables' to nil.  However, this will also disable
> file-local variables.
>
> Perhaps we should add a similar variable `enable-dir-local-variables'
> that will disable only dir-local variables.  It could also have an option
> `remote' to disable searching upwards in remote filesystems.  (This option
> could behave like the variable `dir-locals-chase-remote' in the package
> `dir-locals.el').
>
> --
> Juri Linkov
> http://www.jurta.org/emacs/
>

Re: files.el: Once again impossible to turn off dir-settings

[Tom Tromey]

>>>>> "Juri" == Juri Linkov <juri@jurta.org> writes:

Juri> Perhaps we should add a similar variable
Juri> `enable-dir-local-variables' that will disable only dir-local
Juri> variables.

It sounds like a decent idea to me.

My objection to customizing the file name was based on the (presumed)
social consequences.  IMO there aren't any such consequences to one
user privately disabling a feature.

Tom

Re: files.el: Once again impossible to turn off dir-settings

[Stefan Monnier]

> The slowness --- at least as I observe at my end, has not been
> fixed.
> In the caase of vc-backends, I'm able to avoid global crawls
> across NFS filesystems by setting
> ;;; vc speed up:

> (setq vc-ignore-dir-regexp
> "\\`\\([\\/][\\/]\\|/net/\\|/home/\\|/afs/\\)\\'")
> Notice the addition of /home above  -- in my case /home is nfs
> mounted.

NFS mounting as such is normally not a problem.  So could you explain
exactly how is /home mounted?  Is it an autofs mount?  Do accesses to
/home/foobar automatically trigger access to some network server (even
if /home/foobar doesn't actually exist)?

> I strongly urge you to reconsider forcing all users to  take the
> hit of searching for project settings.

> For now, I've defadviced hack-dir-local-variables like so:
> (defadvice hack-dir-local-variables (around fix-slowness pre act comp)
>   "Restore democracy, restore speed"
> nil)

There's locate-dominating-stop-dir-regexp (which will apply both to VC
and to dir-settings).


        Stefan

files.el: Once again impossible to turn off dir-settings

[T.V. Raman]

Yes, please introduce a customization option that disables
dir-local settings.
   >>>>> "Juri" == Juri Linkov <juri@jurta.org> writes:

   Juri> Perhaps we should add a similar variable
   Juri> `enable-dir-local-variables' that will disable only dir-local
   Juri> variables.

   It sounds like a decent idea to me.

   My objection to customizing the file name was based on the (presumed)
   social consequences. IMO there aren't any such consequences to one
   user privately disabling a feature.

   Tom

-- 
Best Regards,
--raman

Title:  Research Scientist
Email:  raman@google.com
WWW:    http://emacspeak.sf.net/raman/
Google: tv+raman
GTalk:  raman@google.com, tv.raman.tv@gmail.com
PGP:    http://emacspeak.sf.net/raman/raman-almaden.asc



On 11/25/08, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Juri" == Juri Linkov <juri@jurta.org> writes:
>
> Juri> Perhaps we should add a similar variable
> Juri> `enable-dir-local-variables' that will disable only dir-local
> Juri> variables.
>
> It sounds like a decent idea to me.
>
> My objection to customizing the file name was based on the (presumed)
> social consequences.  IMO there aren't any such consequences to one
> user privately disabling a feature.
>
> Tom
>

Re: files.el: Once again impossible to turn off dir-settings

[Chong Yidong]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> (setq vc-ignore-dir-regexp
>> "\\`\\([\\/][\\/]\\|/net/\\|/home/\\|/afs/\\)\\'")
>> Notice the addition of /home above  -- in my case /home is nfs
>> mounted.
>
> NFS mounting as such is normally not a problem.  So could you explain
> exactly how is /home mounted?  Is it an autofs mount?  Do accesses to
> /home/foobar automatically trigger access to some network server (even
> if /home/foobar doesn't actually exist)?

I don't know if this is what T. V. Raman is using, but I have experience
of a setup where moving up the directory hierachy eventually leads to
extreme slowness.  When (say) your home directory is mounted on AFS (a
distributeed network file system), moving up the AFS file hierachy
eventually brings you to the /afs root directory.  This directory is
populated by literally thousands of files, each of which on a different
server (each is a different AFS cell).

Doing something like `ls' in this directory can take minutes.

(I don't use this setup anymore, though; this was on a campus network
where AFS was widely used.)

Re: files.el: Once again impossible to turn off dir-settings

[Stefan Monnier]

>>> (setq vc-ignore-dir-regexp
>>> "\\`\\([\\/][\\/]\\|/net/\\|/home/\\|/afs/\\)\\'")
>>> Notice the addition of /home above  -- in my case /home is nfs
>>> mounted.
>> 
>> NFS mounting as such is normally not a problem.  So could you explain
>> exactly how is /home mounted?  Is it an autofs mount?  Do accesses to
>> /home/foobar automatically trigger access to some network server (even
>> if /home/foobar doesn't actually exist)?

> I don't know if this is what T. V. Raman is using, but I have experience
> of a setup where moving up the directory hierachy eventually leads to
> extreme slowness.  When (say) your home directory is mounted on AFS (a
> distributeed network file system), moving up the AFS file hierachy
> eventually brings you to the /afs root directory.  This directory is
> populated by literally thousands of files, each of which on a different
> server (each is a different AFS cell).

Indeed.  Which is why /afs is handled specially in
locate-dominating-stop-dir-regexp.  With NFS this problem normally
doesn't happen.

> Doing something like `ls' in this directory can take minutes.

dir-settings (and VC) doesn't use the equivalent of `ls' (aka
directory-files).  Instead it looks specifically for the few files that
could matter (e.g. CVS/Entries, .dit-settings.el, ...).  This can make
a very large difference.

> (I don't use this setup anymore, though; this was on a campus network
> where AFS was widely used.)

Yes, I've used AFS as well.  Neat thing (other than Kerberos whose
timeouts I don't like).


        Stefan

files.el: Once again impossible to turn off dir-settings

[T.V. Raman]

Stefane, --yes, /homee/user is nfs mounted -- and accessing
non-existent directories in /home causes a network access. This
means that         when emacs  goes hunting in /home, it triggers
a huge number of filer accesses. I wasn't aware of the other
regexp -- I'll use that. But for the foreseeable future I expect
to disable dir-setings -- at least until there is a project using
it.

I use emacs for many things, including writing code, -- not
exclusively for writing code; and for that use case, it's
important that emacs not always go trawling around for project settings.

-- 
Best Regards,
--raman


Email:  raman@users.sf.net
WWW:    http://emacspeak.sf.net/raman/
AIM:    emacspeak       GTalk: tv.raman.tv@gmail.com
PGP:    http://emacspeak.sf.net/raman/raman-almaden.asc
Google: tv+raman
IRC:    irc://irc.freenode.net/#emacs


On 11/25/08, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> The slowness --- at least as I observe at my end, has not been
>> fixed.
>> In the caase of vc-backends, I'm able to avoid global crawls
>> across NFS filesystems by setting
>> ;;; vc speed up:
>
>> (setq vc-ignore-dir-regexp
>> "\\`\\([\\/][\\/]\\|/net/\\|/home/\\|/afs/\\)\\'")
>> Notice the addition of /home above  -- in my case /home is nfs
>> mounted.
>
> NFS mounting as such is normally not a problem.  So could you explain
> exactly how is /home mounted?  Is it an autofs mount?  Do accesses to
> /home/foobar automatically trigger access to some network server (even
> if /home/foobar doesn't actually exist)?
>
>> I strongly urge you to reconsider forcing all users to  take the
>> hit of searching for project settings.
>
>> For now, I've defadviced hack-dir-local-variables like so:
>> (defadvice hack-dir-local-variables (around fix-slowness pre act comp)
>>   "Restore democracy, restore speed"
>> nil)
>
> There's locate-dominating-stop-dir-regexp (which will apply both to VC
> and to dir-settings).
>
>
>         Stefan
>

Re: files.el: Once again impossible to turn off dir-settings

[Stefan Monnier]

> Stefan, --yes, /homee/user is nfs mounted -- and accessing
> non-existent directories in /home causes a network access.

So if I access /home/<foo>, which remote machine is being accessed?

> means that         when emacs  goes hunting in /home, it triggers
> a huge number of filer accesses.

In my experience, the actual number of accesses is not important.
What matters is whether one of them triggers access to a nonexistent
remote machine.  E.g trying to find /net/.dir-settings.el can cause the
autofs file system to try and contact the ".dir-settings.el" server
which doesn't exist (if you get lucky, you immediately get an error,
but you may also end up waiting for a timeout, and in any case you end
up waiting for DNS resolution, which can take a few seconds).

> I wasn't aware of the other regexp -- I'll use that.  But for the
> foreseeable future I expect to disable dir-setings -- at least until
> there is a project using it.

Sounds reasonable.

> I use Emacs for many things, including writing code, -- not
> exclusively for writing code; and for that use case, it's
> important that emacs not always go trawling around for project settings.

Dir settings are not only for "programming projects".  They can be used
for anything, just like file variables (which I actually tend to use
more in non-programming files).  Also as long as you don't have any
.dir-settings.el files, then the only difference between activating and
deactivating the feature is the time taken to to look up those files,
but since VC looks up several similar files anyway, the slow down should
be lost in the noise (unless you set vc-ignore-dir-regexp to a value
different from locate-dominating-stop-dir-regexp, as you've seen).


        Stefan

Re: files.el: Once again impossible to turn off dir-settings

[Juri Linkov]

> Yes, please introduce a customization option that disables
> dir-local settings.

This is implemented in the following patch:

Index: lisp/files.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/files.el,v
retrieving revision 1.1019
diff -c -r1.1019 files.el
*** lisp/files.el	24 Nov 2008 19:13:58 -0000	1.1019
--- lisp/files.el	27 Nov 2008 00:00:27 -0000
***************
*** 3290,3302 ****
  
  (declare-function c-postprocess-file-styles "cc-mode" ())
  
  (defun hack-dir-local-variables ()
    "Read per-directory local variables for the current buffer.
  Store the directory-local variables in `file-local-variables-alist',
  without applying them."
    (when (and enable-local-variables
  	     (buffer-file-name)
! 	     (not (file-remote-p (buffer-file-name))))
      ;; Find the variables file.
      (let ((variables-file (dir-locals-find-file (buffer-file-name)))
  	  (class nil)
--- 3290,3318 ----
  
  (declare-function c-postprocess-file-styles "cc-mode" ())
  
+ (defcustom enable-dir-local-variables t
+   "Control use of directory-local variables in files you visit.
+ A value of nil means always ignore directory-local variables.  A value
+ of t means obey directory-local variables except when they are on
+ a remote system.  `remote' means search directory-local variables
+ in remote files as well.
+ 
+ File-local customization in variables like `enable-local-variables'
+ is checked in addition to this variable."
+   :type '(choice (const :tag "Enable" t)
+ 		 (const :tag "Remote" remote)
+ 		 (const :tag "Disable" nil))
+   :group 'find-file)
+ 
  (defun hack-dir-local-variables ()
    "Read per-directory local variables for the current buffer.
  Store the directory-local variables in `file-local-variables-alist',
  without applying them."
    (when (and enable-local-variables
+ 	     enable-dir-local-variables
  	     (buffer-file-name)
! 	     (or (eq enable-dir-local-variables 'remote)
! 		 (not (file-remote-p (buffer-file-name)))))
      ;; Find the variables file.
      (let ((variables-file (dir-locals-find-file (buffer-file-name)))
  	  (class nil)

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: files.el: Once again impossible to turn off dir-settings

[Miles Bader]

FWIW, my home dir is in NFS too, and I've noticed no apparently
slowdown.

-Miles

-- 
We are all lying in the gutter, but some of us are looking at the stars.
-Oscar Wilde