Remote display-time-mail-file

Remote display-time-mail-file

[Eli Zaretskii]

Is this supposed to work:

 (setq display-time-mail-file "/plink:eliz@fencepost.gnu.org:/var/mail/eliz")

?

It doesn't work well for me in Emacs 23.2 on MS-Windows, and the
reason seems to be that file-attributes, called by
display-time-file-nonempty-p, only probes the file once, and
thereafter returns the same attributes, even when the file's size
changes.  Is there some cache somewhere?

I tried to debug this, but since I don't even see
tramp-handle-file-attributes being called, I'm probably missing
something very important.  But what?

FWIW, visiting files on the same host via the `plink' method does
work.

TIA

Re: Remote display-time-mail-file

[Eli Zaretskii]

> Date: Thu, 16 Sep 2010 20:38:32 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Is this supposed to work:
> 
>  (setq display-time-mail-file "/plink:eliz@fencepost.gnu.org:/var/mail/eliz")
> 
> ?
> 
> It doesn't work well for me in Emacs 23.2 on MS-Windows, and the
> reason seems to be that file-attributes, called by
> display-time-file-nonempty-p, only probes the file once, and
> thereafter returns the same attributes, even when the file's size
> changes.  Is there some cache somewhere?

The same happens in 3-day old Emacs 24, FWIW, and for the same reason.

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 16 Sep 2010 20:38:32 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> Is this supposed to work:
>> 
>>  (setq display-time-mail-file "/plink:eliz@fencepost.gnu.org:/var/mail/eliz")
>> 
>> ?
>> 
>> It doesn't work well for me in Emacs 23.2 on MS-Windows, and the
>> reason seems to be that file-attributes, called by
>> display-time-file-nonempty-p, only probes the file once, and
>> thereafter returns the same attributes, even when the file's size
>> changes.  Is there some cache somewhere?
>
> The same happens in 3-day old Emacs 24, FWIW, and for the same reason.

In order to check, whether Tramp's cache is involved, you might test the
following patch (it might not be the final solution, 'tho):

--8<---------------cut here---------------start------------->8---
*** /home/albinus/src/emacs/lisp/time.el.~101446~	2010-09-16 22:56:41.003703326 +0200
--- /home/albinus/src/emacs/lisp/time.el	2010-09-16 22:56:09.068503672 +0200
***************
*** 454,461 ****
    (force-mode-line-update))
  
  (defun display-time-file-nonempty-p (file)
!   (and (file-exists-p file)
!        (< 0 (nth 7 (file-attributes (file-chase-links file))))))
  
  ;;;###autoload
  (define-minor-mode display-time-mode
--- 454,462 ----
    (force-mode-line-update))
  
  (defun display-time-file-nonempty-p (file)
!   (let ((tramp-cache-inhibit-cache t))
!     (and (file-exists-p file)
! 	 (< 0 (nth 7 (file-attributes (file-chase-links file)))))))
  
  ;;;###autoload
  (define-minor-mode display-time-mode
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: Remote display-time-mail-file

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel@gnu.org
> Date: Thu, 16 Sep 2010 23:01:35 +0200
> 
> >>  (setq display-time-mail-file "/plink:eliz@fencepost.gnu.org:/var/mail/eliz")
> >> 
> >> ?
> >> 
> >> It doesn't work well for me in Emacs 23.2 on MS-Windows, and the
> >> reason seems to be that file-attributes, called by
> >> display-time-file-nonempty-p, only probes the file once, and
> >> thereafter returns the same attributes, even when the file's size
> >> changes.  Is there some cache somewhere?
> >
> > The same happens in 3-day old Emacs 24, FWIW, and for the same reason.
> 
> In order to check, whether Tramp's cache is involved, you might test the
> following patch (it might not be the final solution, 'tho):

Yes, this fixes the problem, thanks.

So what are the triggers for Tramp to refresh its cache?  Does every
function which deal with files that can be modified behind Emacs's
back need to bind tramp-cache-inhibit-cache to a non-nil value?
That'd be tedious, IMO.

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, this fixes the problem, thanks.
>
> So what are the triggers for Tramp to refresh its cache?  Does every
> function which deal with files that can be modified behind Emacs's
> back need to bind tramp-cache-inhibit-cache to a non-nil value?

Until now, we have only the variable `process-file-side-effects'. When
non-nil, a call of `process-file' is expected to modify files out of
Emacs' control, and packages like Tramp must invalidate the respective
file attributes cache. The default value is t (any process might have
side effects); it could be let-bound to nil in case one knows there
aren't side effects for a given process call.

Apart from this, we have no general mechanism to indicate that files
could have changed. We could introduce a similar variable
`remote-file-side-effects' (tell me a better name, if you know), which
does it say in general. It's default value shall be nil, but in cases
file attributes caches hurt (like in your example), one could let-bind
it to t.

I wouldn't recommend to disable Tramp's file attributes cache at
all. There are serious performance improvements, using the cache.

Best regards, Michael.

Re: Remote display-time-mail-file

[Stefan Monnier]

> I wouldn't recommend to disable Tramp's file attributes cache at
> all. There are serious performance improvements, using the cache.

Isn't there some timeout on the cache entries?


        Stefan

Re: Remote display-time-mail-file

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I wouldn't recommend to disable Tramp's file attributes cache at
>> all. There are serious performance improvements, using the cache.
>
> Isn't there some timeout on the cache entries?

Some internal ones, like `tramp-completion-reread-directory-timeout'. I
could extend it for a general case. I'll check it next days.

>         Stefan

Best regards, Michael.

Re: Remote display-time-mail-file

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>>> I wouldn't recommend to disable Tramp's file attributes cache at
>>> all. There are serious performance improvements, using the cache.
>>
>> Isn't there some timeout on the cache entries?
>
> Some internal ones, like `tramp-completion-reread-directory-timeout'. I
> could extend it for a general case. I'll check it next days.

Maybe it's worth to mention, that `tramp-cache-inhibit-cache' does
something like this:

  tramp-cache-inhibit-cache is a variable defined in `tramp-cache.el'.
  Its value is nil

  Inhibit cache read access, when `t'.
  `nil' means to accept cache entries unconditionally.  If the
  value is a timestamp (as returned by `current-time'), cache
  entries are not used when they have been written before this
  time.

If we use a relative time (number of seconds the cache is valid), and
generalize this variable outside Tramp, it could serve our purposes.

>>         Stefan

Best regards, Michael.

Re: Remote display-time-mail-file

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "emacs-devel\@gnu.org" <emacs-devel@gnu.org>
> Date: Fri, 17 Sep 2010 22:59:36 +0200
> 
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> > Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> >
> >>> I wouldn't recommend to disable Tramp's file attributes cache at
> >>> all. There are serious performance improvements, using the cache.
> >>
> >> Isn't there some timeout on the cache entries?
> >
> > Some internal ones, like `tramp-completion-reread-directory-timeout'. I
> > could extend it for a general case. I'll check it next days.
> 
> Maybe it's worth to mention, that `tramp-cache-inhibit-cache' does
> something like this:
> 
>   tramp-cache-inhibit-cache is a variable defined in `tramp-cache.el'.
>   Its value is nil
> 
>   Inhibit cache read access, when `t'.
>   `nil' means to accept cache entries unconditionally.  If the
>   value is a timestamp (as returned by `current-time'), cache
>   entries are not used when they have been written before this
>   time.
> 
> If we use a relative time (number of seconds the cache is valid), and
> generalize this variable outside Tramp, it could serve our purposes.

Maybe I'm missing something, but I don't see how this can help.  For
starters, time.el needs Tramp to check the remote every time, so the
cache just gets in the way.

For other use-cases, I wonder how can the cache validity be limited in
terms of time.  Files are modified by programs that access them, not
because some amount of time has passed.  How will a Lisp program that
accesses remote files know which number of seconds to set in this
variable?  What am I missing?

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe I'm missing something, but I don't see how this can help.  For
> starters, time.el needs Tramp to check the remote every time, so the
> cache just gets in the way.
>
> For other use-cases, I wonder how can the cache validity be limited in
> terms of time.  Files are modified by programs that access them, not
> because some amount of time has passed.  How will a Lisp program that
> accesses remote files know which number of seconds to set in this
> variable?  What am I missing?

When a remote file is accessed, there are a lot of primitive file name
operations called several times, for example file-attributes. Even if
the timeout is just 10" it will save time, because only the first
file-attributes call needs to go remote.

And if the timeout is customizable, users can decide about the
settings. In my use cases I'm happy with enabled cache all the time.

Best regards, Michael.

Re: Remote display-time-mail-file

[Stefan Monnier]

> Maybe I'm missing something, but I don't see how this can help.  For
> starters, time.el needs Tramp to check the remote every time, so the
> cache just gets in the way.

But each use from time.el might end up calling some internal Tramp
functions several times, so the cache may still speed things up.

> For other use-cases, I wonder how can the cache validity be limited in
> terms of time.  Files are modified by programs that access them, not
> because some amount of time has passed.

Indeed, but short of using some inotify-like thingy, Emacs has no way to
know when a file is modified by some other program, so the best it can
do is to say that it will be out of sync at worst up to N seconds.

> How will a Lisp program that accesses remote files know which number
> of seconds to set in this variable?  What am I missing?

In general, there is no good answer.  But clearly, your time.el case
will want to set this limit to just below to refresh rate.


        Stefan

Re: Remote display-time-mail-file

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Indeed, but short of using some inotify-like thingy, Emacs has no way to
> know when a file is modified by some other program, so the best it can
> do is to say that it will be out of sync at worst up to N seconds.

A while ago, I was playing with the idea to install an inotify listener
on the remote host. I gave it up, because I haven't found a reasonable
filter which files shall be reported as being modified.

>         Stefan

Best regards, Michael.

Re: Remote display-time-mail-file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Michael Albinus <michael.albinus@gmx.de>, emacs-devel@gnu.org
> Date: Sat, 18 Sep 2010 11:00:27 +0200
> 
> > Maybe I'm missing something, but I don't see how this can help.  For
> > starters, time.el needs Tramp to check the remote every time, so the
> > cache just gets in the way.
> 
> But each use from time.el might end up calling some internal Tramp
> functions several times, so the cache may still speed things up.

That's true, but my conclusion from this is different: we probably
should have finer granularity of the cache setting.

I guess some of the internal Tramp functions need not be called any
time soon after the first call, for example those that find out which
method to use to access the remote and which scripts to run on the
remote side.  Other internal functions will need to run every time
display-time-file-nonempty-p is called.  And there could be those in
between.  So perhaps these internal functions should be categorized in
some reasonable manner, and then corresponding values added to the
repertoire of tramp-cache-inhibit-cache's values, so that Lisp
programs could have finer control on what is being cached and when the
cache is refreshed.

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> That's true, but my conclusion from this is different: we probably
> should have finer granularity of the cache setting.
>
> I guess some of the internal Tramp functions need not be called any
> time soon after the first call, for example those that find out which
> method to use to access the remote and which scripts to run on the
> remote side.  Other internal functions will need to run every time
> display-time-file-nonempty-p is called.  And there could be those in
> between.  So perhaps these internal functions should be categorized in
> some reasonable manner, and then corresponding values added to the
> repertoire of tramp-cache-inhibit-cache's values, so that Lisp
> programs could have finer control on what is being cached and when the
> cache is refreshed.

When I have started with Tramp 2.1, 5 years ago, I did some profiling
with Tramp. IIRC, there are only some few functions (like
file-attributes), which are called frequently, and which profit from
caching more than other functions. Maybe we shall rerun this profiling
again, since we have started Tramp 2.2 :-)

I guess we could use caching just for these high-runner functions; for
the other ones it doesn't matter whether we have caching, or not.

Best regards, Michael.

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>> Cc: Michael Albinus <michael.albinus@gmx.de>, emacs-devel@gnu.org
>> Date: Sat, 18 Sep 2010 11:00:27 +0200
>>
>> > Maybe I'm missing something, but I don't see how this can help.  For
>> > starters, time.el needs Tramp to check the remote every time, so the
>> > cache just gets in the way.
>>
>> But each use from time.el might end up calling some internal Tramp
>> functions several times, so the cache may still speed things up.
>
> That's true, but my conclusion from this is different: we probably
> should have finer granularity of the cache setting.
>
> I guess some of the internal Tramp functions need not be called any
> time soon after the first call, for example those that find out which
> method to use to access the remote and which scripts to run on the
> remote side.  Other internal functions will need to run every time
> display-time-file-nonempty-p is called.  And there could be those in
> between.  So perhaps these internal functions should be categorized in
> some reasonable manner, and then corresponding values added to the
> repertoire of tramp-cache-inhibit-cache's values, so that Lisp
> programs could have finer control on what is being cached and when the
> cache is refreshed.

I did some tests. It is hard to predict, which primitive functions are
called where, and in majority thes are file-attributes, file-exists-p,
file-readable-p, file-writable-p. Other primitive functions do profit
from those cached values, or they aren't called as much that it is worth
to spend extra handling for caching.

Therefore, I propose the following patch:

--8<---------------cut here---------------start------------->8---
*** /home/albinus/src/emacs/lisp/files.el.~101670~	2010-09-30 15:02:06.590646221 +0200
--- /home/albinus/src/emacs/lisp/files.el	2010-09-30 14:24:08.655019756 +0200
***************
*** 934,939 ****
--- 934,968 ----
  	(funcall handler 'file-remote-p file identification connected)
        nil)))

+ (defcustom remote-file-name-inhibit-cache 10
+   "Inhibit remote file name cache read access, when `t'.
+ `nil' means to accept cache entries unconditionally.  An integer
+ prevents using of cache entries if they are older than this
+ amount of seconds.
+
+ File attributes of remote files are cached for better performance.
+ If they are changed out of Emacs' control, the cached values
+ become invalid, and must be invalidated.
+
+ In case a remote file is checked regularly, it might be
+ reasonable to let-bind this variable to a value less then the
+ time period between two checks.
+
+ Example:
+
+   \(defun display-time-file-nonempty-p \(file)
+     \(let \(\(remote-file-name-inhibit-cache \(- display-time-interval 5)))
+       \(and \(file-exists-p file)
+            \(< 0 \(nth 7 \(file-attributes \(file-chase-links file)))))))"
+   :group 'files
+   :version "24.1"
+   :type `(choice
+ 	  (const   :tag "Do not inhibit file name cache" nil)
+ 	  (const   :tag "Do not use file name cache" t)
+ 	  (integer :tag "Do not use file name cache"
+ 		   :format "Do not use file name cache older then %v seconds"
+ 		   :value 10)))
+
  (defun file-local-copy (file)
    "Copy the file FILE into a temporary file on this machine.
  Returns the name of the local copy, or nil, if FILE is directly
--8<---------------cut here---------------end--------------->8---

Tramp would be adapted accordingly, of course.

Best regards, Michael.

Re: Remote display-time-mail-file

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org
> Date: Thu, 30 Sep 2010 15:07:34 +0200
> 
> I did some tests. It is hard to predict, which primitive functions are
> called where, and in majority thes are file-attributes, file-exists-p,
> file-readable-p, file-writable-p. Other primitive functions do profit
> from those cached values, or they aren't called as much that it is worth
> to spend extra handling for caching.
> 
> Therefore, I propose the following patch:

Looks good, thanks.  One suggestion for a more clear doc string:

(defcustom remote-file-name-inhibit-cache 10
  "Whether to use the remote file-name cache for read access.

When `nil', always use the cached values.
When `t', never use them.
A number means use them for that amount of seconds since they were
cached.

File attributes of remote files are cached for better performance.
If they are changed out of Emacs' control, the cached values
become invalid, and must be invalidated.

In case a remote file is checked regularly, it might be
reasonable to let-bind this variable to a value less then the
time period between two checks.
Example:

  \(defun display-time-file-nonempty-p \(file)
    \(let \(\(remote-file-name-inhibit-cache \(- display-time-interval 5)))
      \(and \(file-exists-p file)
           \(< 0 \(nth 7 \(file-attributes \(file-chase-links file)))))))"

Re: Remote display-time-mail-file

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Looks good, thanks.  One suggestion for a more clear doc string:

Thanks, I'm always happy if somebody improves my English ... If nobody
objects, I'll install it over the weekend in the trunk.

Best regards, Michael.