Optional argument for `file-local-copy'

Optional argument for `file-local-copy'

[Michael Albinus]

Hi,

I would like to introduce an optional argument REUSE-IF-ALREADY-EXISTS
for `file-local-copy'. The docstring would be something like this:

--8<---------------cut here---------------start------------->8---
file-local-copy is a compiled Lisp function in `files.el'.

(file-local-copy FILE &optional REUSE-IF-ALREADY-EXISTS)

Copy the file FILE into a temporary file on this machine.
Returns the name of the local copy, or nil, if FILE is directly
accessible.

If REUSE-IF-ALREADY-EXISTS is non-nil, if there exists already a local
copy of FILE, and if the file attributes of FILE have not been changed
since the last copy operation, the name of the existing local copy of
FILE will be returned.
--8<---------------cut here---------------end--------------->8---

A typical use case for this has been discussed on the org-mode ML
<http://thread.gmane.org/gmane.emacs.orgmode/92856/focus=92970>. The
idea there is to cache downloaded images. But this parameter could be
used for caching copies of remote files in general.

Comments?

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Sat, 29 Nov 2014 12:29:21 +0100
> 
> file-local-copy is a compiled Lisp function in `files.el'.
> 
> (file-local-copy FILE &optional REUSE-IF-ALREADY-EXISTS)
> 
> Copy the file FILE into a temporary file on this machine.
> Returns the name of the local copy, or nil, if FILE is directly
> accessible.
> 
> If REUSE-IF-ALREADY-EXISTS is non-nil, if there exists already a local
> copy of FILE, and if the file attributes of FILE have not been changed
> since the last copy operation, the name of the existing local copy of
> FILE will be returned.

"If" within another "if" is considered harmful.  Suggested rewording:

  REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
  local copy of FILE, if FILE's file attributes didn't change since
  the last copy operation.

(I have no opinion on the rest of your proposal.)

Re: Optional argument for `file-local-copy'

[Alan Mackenzie]

Hello, Eli and Michael.

On Sat, Nov 29, 2014 at 03:03:31PM +0200, Eli Zaretskii wrote:
> > From: Michael Albinus <michael.albinus@gmx.de>
> > Date: Sat, 29 Nov 2014 12:29:21 +0100

> > file-local-copy is a compiled Lisp function in `files.el'.

> > (file-local-copy FILE &optional REUSE-IF-ALREADY-EXISTS)

> > Copy the file FILE into a temporary file on this machine.
> > Returns the name of the local copy, or nil, if FILE is directly
> > accessible.

> > If REUSE-IF-ALREADY-EXISTS is non-nil, if there exists already a local
> > copy of FILE, and if the file attributes of FILE have not been changed
> > since the last copy operation, the name of the existing local copy of
> > FILE will be returned.

> "If" within another "if" is considered harmful.  Suggested rewording:

>   REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
>   local copy of FILE, if FILE's file attributes didn't change since
                                                  ^^^^^^^^^^^^^
>   the last copy operation.

"haven't changed", here, since it's expressing a current state.  "didn't
change" would be appropriate for referring to a past event.

> (I have no opinion on the rest of your proposal.)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Optional argument for `file-local-copy'

[Eli Zaretskii]

> Date: Sat, 29 Nov 2014 14:08:57 +0000
> Cc: Michael Albinus <michael.albinus@gmx.de>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> >   REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
> >   local copy of FILE, if FILE's file attributes didn't change since
>                                                   ^^^^^^^^^^^^^
> >   the last copy operation.
> 
> "haven't changed", here, since it's expressing a current state.  "didn't
> change" would be appropriate for referring to a past event.

It _is_ a past event, by the time this function is called.

Re: Optional argument for `file-local-copy'

[Alan Mackenzie]

Hi, Eli.

On Sat, Nov 29, 2014 at 04:39:47PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 29 Nov 2014 14:08:57 +0000
> > Cc: Michael Albinus <michael.albinus@gmx.de>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > >   REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
> > >   local copy of FILE, if FILE's file attributes didn't change since
> >                                                   ^^^^^^^^^^^^^
> > >   the last copy operation.

> > "haven't changed", here, since it's expressing a current state.  "didn't
> > change" would be appropriate for referring to a past event.

> It _is_ a past event, by the time this function is called.

Yes, but it is that event's effect on the present which makes the
"present perfect" the right tense.

"haven't changed" is the correct tense here.  Really it is.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Optional argument for `file-local-copy'

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

() Alan Mackenzie <acm@muc.de>
() Sat, 29 Nov 2014 15:33:20 +0000

   > > > REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
   > > > local copy of FILE, if FILE's file attributes didn't change since
   > >                                                 ^^^^^^^^^^^^^
   > > > the last copy operation.

   > > "haven't changed", here, since it's expressing a current state.  "didn't
   > > change" would be appropriate for referring to a past event.

   > It _is_ a past event, by the time this function is called.

   Yes, but it is that event's effect on the present which makes
   the "present perfect" the right tense.

   "haven't changed" is the correct tense here.  Really it is.

Another formulation is "if FILE's file attributes are unchanged".
This shifts the emphasis from the interval to the present, which
is the "time" i think "in" for trying to grok these things.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

> "haven't changed" is the correct tense here.  Really it is.

Agreed.  As always when used with "since",


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Eli.

Hi Eli, Alan,

>> > >   REUSE-IF-ALREADY-EXISTS non-nil means return the name of an existing
>> > >   local copy of FILE, if FILE's file attributes didn't change since
>> >                                                   ^^^^^^^^^^^^^
>> > >   the last copy operation.
>
>> > "haven't changed", here, since it's expressing a current state.  "didn't
>> > change" would be appropriate for referring to a past event.
>
>> It _is_ a past event, by the time this function is called.
>
> Yes, but it is that event's effect on the present which makes the
> "present perfect" the right tense.
>
> "haven't changed" is the correct tense here.  Really it is.

Thanks for improving the docstring! I'm not a native speaker, so it is
always helpful for me.

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> "haven't changed" is the correct tense here.  Really it is.
>
> Agreed.  As always when used with "since",

Does "Agreed" means also you agree with the idea of that optional argument?

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

> Does "Agreed" means also you agree with the idea of that optional argument?

Thanks for getting us back on track.  Yes, I agree with adding such
a functionality.  But I have the impression that it's not enough: if the
caller doesn't know whether the file is indeed being cached or not, then
it doesn't know whether to delete it or not once it's done using it.



        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Thanks for getting us back on track.  Yes, I agree with adding such
> a functionality.  But I have the impression that it's not enough: if the
> caller doesn't know whether the file is indeed being cached or not, then
> it doesn't know whether to delete it or not once it's done using it.

The caller does not need to know whether a file is cached. File name
handlers implementation shall keep *always* the information about a
local copy of a remote file (the local name, the result of
file-attributes of the remote file).

When file-local-copy is called with a non-nil REUSE-IF-ALREADY-EXISTS,
it checks whether there is a cached information about that remote file,
whether the local copy still exists, and whether the cached
file-attributes information is equal to the actual value. Then, and only
then, it reuses that local copy.

By this, any caller of file-local-copy could either ignore this new
argument, and delete the local copy after it was used, or the caller
does not delete the local copy because there is the assumption that the
local copy might be used, again. Candidates are retrieved web pages, or
remote files retrieved by a vc backend, like vc-bzr-admin-* files in
vc-bzr.el.

A problem might be to remove local copies when they are not used
anymore. This could be done when Emacs is stopped, and/or when Emacs is
started. But this shall be applied in the background, without
intervention of the user.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

>> Thanks for getting us back on track.  Yes, I agree with adding such
>> a functionality.  But I have the impression that it's not enough: if the
>> caller doesn't know whether the file is indeed being cached or not, then
>> it doesn't know whether to delete it or not once it's done using it.
> The caller does not need to know whether a file is cached.  File name
> handlers implementation shall keep *always* the information about a
> local copy of a remote file (the local name, the result of
> file-attributes of the remote file).

That means that the file-local-copy handlers *always* have to obey
REUSE-IF-ALREADY-EXISTS, rather only doing so if they use a cache.

> A problem might be to remove local copies when they are not
> used anymore.

That's inherent in the use of a cache, I'm not too worried about that,
tho of course, we'll have to make it work well (erasing at the end of
the session is problematic since the session might not end cleanly, so
we'll probably want to instead make the cache survive for longer, and
share it across sessions).


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The caller does not need to know whether a file is cached.  File name
>> handlers implementation shall keep *always* the information about a
>> local copy of a remote file (the local name, the result of
>> file-attributes of the remote file).
>
> That means that the file-local-copy handlers *always* have to obey
> REUSE-IF-ALREADY-EXISTS, rather only doing so if they use a cache.

Well, it does not need to obey that argument. My idea is that the
*meta information* are always stored.

>> A problem might be to remove local copies when they are not
>> used anymore.
>
> That's inherent in the use of a cache, I'm not too worried about that,
> tho of course, we'll have to make it work well (erasing at the end of
> the session is problematic since the session might not end cleanly, so
> we'll probably want to instead make the cache survive for longer, and
> share it across sessions).

Yes. This might be mainly problem when the cache is too small on the
file system. Maybe one could restrict the size of the files being
cacheable. But OTOH, very large files profit more from being cached.

Another idea could be to implement a timeout for cached files; after a
given time they are not valid anymore. But that's not needed in the
beginning; I would add some functionality when there's a need from the
users.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

>>> The caller does not need to know whether a file is cached.  File name
>>> handlers implementation shall keep *always* the information about a
>>> local copy of a remote file (the local name, the result of
>>> file-attributes of the remote file).
>> That means that the file-local-copy handlers *always* have to obey
>> REUSE-IF-ALREADY-EXISTS, rather only doing so if they use a cache.
> Well, it does not need to obey that argument.

No: if the handler ignores the argument and the caller doesn't delete
the file, we're left with a garbage file that noone will delete.


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> No: if the handler ignores the argument and the caller doesn't delete
> the file, we're left with a garbage file that noone will delete.

Like today.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

>> No: if the handler ignores the argument and the caller doesn't delete
>> the file, we're left with a garbage file that noone will delete.
> Like today.

There's a big difference: with today's file-local-copy the caller is
always responsible to delete the file, so if the caller doesn't delete
the file, it's a bug in the caller.

With your proposal, if it's not the responsibility of the caller, then
it necessarily ends up being the handler's responsibility.  Hence my
earlier complaint that the handler is not allowed to simply ignore the
new argument.


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[sorry to reply late, but these days there isn't too much time I could
spend for Emacs]

> There's a big difference: with today's file-local-copy the caller is
> always responsible to delete the file, so if the caller doesn't delete
> the file, it's a bug in the caller.
>
> With your proposal, if it's not the responsibility of the caller, then
> it necessarily ends up being the handler's responsibility.  Hence my
> earlier complaint that the handler is not allowed to simply ignore the
> new argument.

No, it is still the caller's responsibility to delete the file. The new
argument is not intended to move this responsibility to the handler; I
expect the caller to keep the file only for a while when it is expecting
that a given file is needed several times (like meta information if a file
in vc).

Additional functionality (remove local copies at the end of an Emacs
session; give the local copies a timeout after which they expire, and so
on) should be only additional support.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

> No, it is still the caller's responsibility to delete the file.

Ah, that definitely was not clear until now.  So the handler never
needs to track those files.

> I expect the caller to keep the file only for a while when it is
> expecting that a given file is needed several times (like meta
> information if a file in vc).

I'm wondering how often the caller will be able to make use of this
(i.e. how often it will happen that two or more calls to file-local-copy
will require the same file and will know enough about each other to
coordinate the file-deletion).

Do you have some callers in mind?

> Additional functionality (remove local copies at the end of an Emacs
> session; give the local copies a timeout after which they expire, and so
> on) should be only additional support.

Right, and this support will not be used on the handlers's side but on
the callers's side.


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I expect the caller to keep the file only for a while when it is
>> expecting that a given file is needed several times (like meta
>> information if a file in vc).
>
> I'm wondering how often the caller will be able to make use of this
> (i.e. how often it will happen that two or more calls to file-local-copy
> will require the same file and will know enough about each other to
> coordinate the file-deletion).
>
> Do you have some callers in mind?

The first idea to this came when I saw
<http://thread.gmane.org/gmane.emacs.orgmode/92856>. It offered a patch
to keep some copied remote images in a cache in order to redisplay
them.

Another evidence for such functionality is vc. If you open a remote file
under bzr control, there will be local copies of .bzr/checkout/dirstate,
.bzr/branch/format and .bzr/branch/last-revision. In the cvs case,
CVS/Entries is taken via file-local-copy. And there might be similar
cases for the other vc backends, I haven't checked. All of them are
candidates for a file cache as proposed.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

> The first idea to this came when I saw
> <http://thread.gmane.org/gmane.emacs.orgmode/92856>. It offered a patch
> to keep some copied remote images in a cache in order to redisplay
> them.

> Another evidence for such functionality is vc. If you open a remote file
> under bzr control, there will be local copies of .bzr/checkout/dirstate,
> .bzr/branch/format and .bzr/branch/last-revision. In the cvs case,
> CVS/Entries is taken via file-local-copy. And there might be similar
> cases for the other vc backends, I haven't checked. All of them are
> candidates for a file cache as proposed.

Yes, but how will you write the code so that the temp/cached files
don't linger indefinitely?

Don't get me wrong: I like the idea, but I think we need to think a bit
more about how to handle the cache and deletion, in order to design the
API correctly.

Here's the problem I currently see:
- OT1H you currently have the caller as the one responsible for the
  deletion of the file.
- OTOH it's the handler that decides where to put the file.
That seems to make it difficult to get a good behavior.  E.g. you might
want to keep the file for a long time (across sessions) in a special
"cache" subdirectory somewhere, but this decision can't be made by the
handler (because the caller will have to delete the file without knowing
it's meant to be kept) and it can't be made by the caller either
(because it can't tell the handler where to put the file).

Maybe a better option is to change the optional argument so that it's
not just a boolean but it's actually the name of a "candidate local
copy" (i.e. it's the name of the local copy kept in the cache managed
by the caller), tho maybe we'd need more info (e.g. the full
file-attributes).


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Yes, but how will you write the code so that the temp/cached files
> don't linger indefinitely?
>
> Don't get me wrong: I like the idea, but I think we need to think a bit
> more about how to handle the cache and deletion, in order to design the
> API correctly.
>
> Here's the problem I currently see:
> - OT1H you currently have the caller as the one responsible for the
>   deletion of the file.
> - OTOH it's the handler that decides where to put the file.
> That seems to make it difficult to get a good behavior.  E.g. you might
> want to keep the file for a long time (across sessions) in a special
> "cache" subdirectory somewhere, but this decision can't be made by the
> handler (because the caller will have to delete the file without knowing
> it's meant to be kept) and it can't be made by the caller either
> (because it can't tell the handler where to put the file).

Currently, `file-local-copy' keeps the files in `temporary-file-directory'.
One could use a subdirectory of that.

And I don't know whether such local copies shall be kept over Emacs
session boundaries. In the Tramp case, it might be acceptable to remove
all local file copies for a given remote host, when the connection to
that host is (re-)established.

> Maybe a better option is to change the optional argument so that it's
> not just a boolean but it's actually the name of a "candidate local
> copy" (i.e. it's the name of the local copy kept in the cache managed
> by the caller), tho maybe we'd need more info (e.g. the full
> file-attributes).

The mapping of a remote file name to the local copy could be managed
inside the `file-local-copy' machinery. And whether we always need
file-attributes I'm not sure. One implementation inside Tramp could be
to install a file notifications handler for the remote file if possible.
We will be informed then, when the remote file changes, and the local
file is out-dated.

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

> file-attributes I'm not sure. One implementation inside Tramp could be
> to install a file notifications handler for the remote file if possible.
> We will be informed then, when the remote file changes, and the local
> file is out-dated.

Right, I guess for a client-side cache, the issue is mostly how to check
the freshness of a file, and it seems difficult to design an API that
can cleanly handle all the various ways we could do that.

So maybe the better option is to do the caching on the handler-side.
But there's no need for any changes to file-local-copy for that.


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> So maybe the better option is to do the caching on the handler-side.
> But there's no need for any changes to file-local-copy for that.

Do you mean we don't need the REUSE-IF-ALREADY-EXISTS argument, and
file-local-copy shall always cache?

>         Stefan

Best regards, Michael.

Re: Optional argument for `file-local-copy'

[Stefan Monnier]

>> So maybe the better option is to do the caching on the handler-side.
>> But there's no need for any changes to file-local-copy for that.
> Do you mean we don't need the REUSE-IF-ALREADY-EXISTS argument, and
> file-local-copy shall always cache?

Right: the handler downloads into the cache, and then serves a copy from
the cache (so file-local-copy would always return a new file, but the
copy is local).


        Stefan

Re: Optional argument for `file-local-copy'

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Right: the handler downloads into the cache, and then serves a copy from
> the cache (so file-local-copy would always return a new file, but the
> copy is local).

OK, I'll implement this in Tramp next days. Afterwards, I'll check where
to apply, for example in vc backends.

>         Stefan

Best regards, Michael.