Dired C idea

Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What would people think about making the C command in Dired
run rsync to do the copying of a directory>

The idea is that interrupting the copying would let you
restart it without starting from zero.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Alexandre Garreau]

Le samedi 31 juillet 2021, 03:34:12 CEST Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> What would people think about making the C command in Dired
> run rsync to do the copying of a directory>
> 
> The idea is that interrupting the copying would let you
> restart it without starting from zero.

sounds really cool and useful (especially as it would just as well allow 
to seemlessly copy to remote), but makes wonder what is cp for nowadays we 
have rsync?

just in case cp still has something not in rsync, shouldn’t we ask the 
coreutils people (current maintainer?) what could it be, so we’d know as 
well it would be neat to integrate it/them into rsync so the decision of 
switching from cp to rsync would be purely positive in any respect (and 
maybe also other places as well)

Re: Dired C idea

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

Sounds like something worth evaluating, but wonder if....

Is rsync available on all supported platforms?

Is rsync installed by default on supported platforms or will this mean that a
core operation i.e. dired, will require installation of rsync before Emacs will
work correctly?

Are there any issues needing to be considered wrt rsync and different file
systems i.e. btrfs, zfs, etc.

Should we replace cp with rsync or should we add rsync as an option/additional
command? Not sure what the impact of rsync might be with simpler cp operations.

Regards,

Tim

--
*Tim Cross*

/For gor sake stop laughing, this is serious!/

Re: Dired C idea

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Sat, 31 Jul 2021 13:35:39 +1000
> 
> Is rsync available on all supported platforms?

We can test that before invoking the command.  We do that in many
other cases.  So this issue is not serious.

Re: Dired C idea

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Date: Sat, 31 Jul 2021 13:35:39 +1000
>> 
>> Is rsync available on all supported platforms?
>
> We can test that before invoking the command.  We do that in many
> other cases.  So this issue is not serious.

It is already possible to run rsync form emacs, by simply calling it
from lisp code. I do it all the time :)

Why do you need it in hardcoded in C code? What extra does it offer?

There are like different versions of ellisp already, most of them spun
from this blog post:

https://truongtx.me/tmtxt-dired-async.html

https://github.com/stsquad/dired-rsync
https://oremacs.com/2016/02/24/dired-rsync/
https://gist.github.com/l3msh0/5006645

Re: Dired C idea

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 31 Jul 2021 18:03:35 +0200
> 
> Why do you need it in hardcoded in C code?

This is a misunderstanding, I think: Richard said "C command" meaning
the command bound to "C" in Dired mode.  He didn't mean to hardcode
this in C code.

Re: Dired C idea

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
>> Date: Sat, 31 Jul 2021 18:03:35 +0200
>> 
>> Why do you need it in hardcoded in C code?
>
> This is a misunderstanding, I think: Richard said "C command" meaning
> the command bound to "C" in Dired mode.  He didn't mean to hardcode
> this in C code.

Ah, sorry, indeed it is misundestanding on my part. I appologize.

RE: [External] : Re: Dired C idea

[Drew Adams]

> Should we replace cp with rsync or should we add rsync as an
> option/additional command? Not sure what the impact of rsync
> might be with simpler cp operations.

How about adding it, and if it turns out to be good
then give it a different key binding, leaving `C' as
it is?

But perhaps I don't understand the suggestion well.
If this behavior is added, is there really no longer
any use for the current `C' behavior?  IOW, why would
this _replace_ that?

Re: [External] : Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But perhaps I don't understand the suggestion well.
  > If this behavior is added, is there really no longer
  > any use for the current `C' behavior?

I see no reason to prefer ordinary copying to rsync.
If you see a reason, could you tell us what it is?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : Re: Dired C idea

[Drew Adams]

>> But perhaps I don't understand the suggestion well.
>> If this behavior is added, is there really no longer
>> any use for the current `C' behavior?
> 
> I see no reason to prefer ordinary copying to rsync.

Ever?  Or do you see any relative advantages to each?

> If you see a reason, could you tell us what it is?

I didn't say anything about preferring ordinary
copying.

Is there a reason that Unix or GNU/Linux `cp' still
exists (if it does)?  Would that reason be relevant
here (just asking)?

All I asked was why replace the command bound to
`C'.  Why not add rsync copying and (if it deserves
a binding) give it a different key from `C'?

I'm not arguing against adding rsync copying, or
against binding it to a key, or even giving it the
`C' binding.  I only asked why it should be given
the `C' binding.

We didn't give async shell commands the `!' Dired
binding.  We instead added the `&' binding.  If
you feel this is different, OK; what are the
reasons for that?

Are you also proposing to remove ordinary copying
altogether?  If not, then the only question (so
far) is about key bindings.

Re: [External] : Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > All I asked was why replace the command bound to
  > `C'.

Because that's the Dired command to copy files.
I think of rsync as a better way of copying them.
So I think that the default way to do the copying.

When you copy individual files in Dired, it tells you which
have been copied.  That's different from rsync but it achieves
the same purpose.

But in recursive copying of a directory, Dired does not
help you avoid repeated copying when you restart.

           Why not add rsync copying and (if it deserves
  > a binding) give it a different key from `C'?

I don't see a need to have a binding for cp.

I did not know about the command copy-directory, but making that
use rsync seems like a good idea.

I recall that rsync has a bug in handling arguments that contain
spaces.  We would need to get that fixed, in order for this change not
to cause problems.

  > Are you also proposing to remove ordinary copying
  > altogether?

I'm not sure what "altogether" means here, since there are
several levels for how far we could go.

I see no need for Dired to have a key to do cp rather than rsync, when
operating on ordinary local disks.

Someone said it might be better not to use rsync when the files are
accessed through Tramp.  I don't know -- I don't use Tramp.
But I agree we should DTRT about that case.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : Re: Dired C idea

[Drew Adams]

>   > All I asked was why replace the command bound to `C'.
> 
> Because that's the Dired command to copy files.
> I think of rsync as a better way of copying them.
> So I think that the default way to do the copying.
> 
> When you copy individual files in Dired, it tells you which
> have been copied.  That's different from rsync but it achieves
> the same purpose.
> 
> But in recursive copying of a directory, Dired does not
> help you avoid repeated copying when you restart.
> 
>            Why not add rsync copying and (if it deserves
>   > a binding) give it a different key from `C'?
> 
> I don't see a need to have a binding for cp.
> 
> I did not know about the command copy-directory, but making that
> use rsync seems like a good idea.
> 
> I recall that rsync has a bug in handling arguments that contain
> spaces.  We would need to get that fixed, in order for this change not
> to cause problems.
> 
>   > Are you also proposing to remove ordinary copying
>   > altogether?
> 
> I'm not sure what "altogether" means here, since there are
> several levels for how far we could go.
> 
> I see no need for Dired to have a key to do cp rather than rsync, when
> operating on ordinary local disks.
> 
> Someone said it might be better not to use rsync when the files are
> accessed through Tramp.  I don't know -- I don't use Tramp.
> But I agree we should DTRT about that case.

Thanks for making your position clear - clear answers.

Re: [External] : Re: Dired C idea

[Fabrice Bauzac-Stehly]

Richard Stallman <rms@gnu.org> writes:

>   >                 why replace the command bound to
>   > `C'

> Because that's the Dired command to copy files.
> I think of rsync as a better way of copying them.

Currently, "C" asks before overwriting files.  rsync overwrites files
without asking.  Users who are used to having a prompt to prevent
accidental overwrites could be surprised of the new behaviour.  Maybe it
could be gentler to allow both cp and rsync implementations, and ask for
which to use on the first invocation (prominently indicating the new
overwrite behaviour), saving the setting for all later uses.

Which options should we give to rsync by default?  Maybe:

  -a --hard-links --partial --one-file-system

I'm not sure whether we should add --acls and/or --xattrs.

> I recall that rsync has a bug in handling arguments that contain
> spaces.  We would need to get that fixed, in order for this change not
> to cause problems.

I think that's what --protect-args do.  Let's add it to the default
options.

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: [External] : Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Currently, "C" asks before overwriting files.  rsync overwrites files
  > without asking.

This looks like a sharp difference between the cp and rsync, but I
suspect this compares cp in one scenario with rsync in another usage
scenario.  It is hard to see this because the statement is so terse,
and I can't be sure of it, but I think it is so.

If you describe the two usage scenarios, we could tell for certain.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Dired C idea

[Fabrice Bauzac-Stehly]

Richard Stallman <rms@gnu.org> writes:

>   > Currently, "C" asks before overwriting files.  rsync overwrites files
>   > without asking.
>
> This looks like a sharp difference between the cp and rsync, but I
> suspect this compares cp in one scenario with rsync in another usage
> scenario.  It is hard to see this because the statement is so terse,
> and I can't be sure of it, but I think it is so.
>
> If you describe the two usage scenarios, we could tell for certain.

Here is the usage scenario I'm thinking about:

Let's imagine a user with a scanner which generates a time-based name
for each pdf file it scans.  That person regularly scans per invoices
and uses the "C" command to archive them into directory "invoices/",
while changing the name at the same time to "invoice-NiceCompany.pdf".

It sometimes happens that this person forgets that a long time ago a
file with the same name was already saved, for another invoice.  This
person trusts the "C" command to tell if an overwrite is going to
happen, preventing the loss of the old invoice.

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: [External] : Re: Dired C idea

[Fabrice Bauzac-Stehly]

Fabrice Bauzac-Stehly <noon@mykolab.com> writes:

> Let's imagine a user with a scanner which generates a time-based name
> for each pdf file it scans.  That person regularly scans per invoices
> and uses the "C" command to archive them into directory "invoices/",
> while changing the name at the same time to "invoice-NiceCompany.pdf".

On second thought, in this case, I think the user would use R rather
than C.

The dangerous behaviour seems much more theoritical to me now.  Sorry
for the noise...

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: [External] : Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Let's imagine a user with a scanner which generates a time-based name
  > for each pdf file it scans.  That person regularly scans per invoices
  > and uses the "C" command to archive them into directory "invoices/",
  > while changing the name at the same time to "invoice-NiceCompany.pdf".

Would you please describe the commands more precisely?  Are you using
the C command on just one file at a time?  Or on several files at a
time?  Or on a directory for recursive copying?

I'm talking about the case of recursive copying of a directory.

  > It sometimes happens that this person forgets that a long time ago a
  > file with the same name was already saved, for another invoice.  This
  > person trusts the "C" command to tell if an overwrite is going to
  > happen, preventing the loss of the old invoice.

I'm thinking about the case of copying a directory.  With the C
command as it is now, you can copy a directory to make a new
directory.  I don't think it is possible to use it to merge a
directory's files into an existing directory.

We could add the ability to do that -- it would be natural if it
worked using rsync -- but that is a case not currently supported.
So there is no possibility of an incompatible change in that case.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Dired C idea

[Fabrice Bauzac-Stehly]

Richard Stallman <rms@gnu.org> writes:

> Are you using the C command on just one file at a time?  Or on several
> files at a time?  Or on a directory for recursive copying?

It would be one file at a time.

> I'm thinking about the case of copying a directory.  With the C
> command as it is now, you can copy a directory to make a new
> directory.  I don't think it is possible to use it to merge a
> directory's files into an existing directory.
>
> We could add the ability to do that -- it would be natural if it
> worked using rsync -- but that is a case not currently supported.
> So there is no possibility of an incompatible change in that case.

Indeed!

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> What would people think about making the C command in Dired
> run rsync to do the copying of a directory>
>
> The idea is that interrupting the copying would let you
> restart it without starting from zero.

Internally, `copy-directory' is called, which is aware of file name
handlers. Using rsync instead of copy shouldn't break this mechanism.

If a remote directory is involved, either source or target, it is
already possible to use Tramp's `rsync' method.

Best regards, Michael.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Internally, `copy-directory' is called, which is aware of file name
  > handlers. Using rsync instead of copy shouldn't break this mechanism.

Should `copy-directory' itself, on local directories, run rsync?
Maybe that would be a simple way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > Internally, `copy-directory' is called, which is aware of file name
>   > handlers. Using rsync instead of copy shouldn't break this mechanism.
>
> Should `copy-directory' itself, on local directories, run rsync?
> Maybe that would be a simple way.

Should be feasible. The hard part is a rule to determine, whether rsync
is better suited than the existing implementation, with the given actual
arguments. I suspect, that small directories and/or short files are
better served with the current implementation.

There exists also the librsync library, released under GNU LGPL v2.1. It
might help to implement rsync in Emacs core, instead of calling an
external process.

Best regards, Michael.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Should be feasible. The hard part is a rule to determine, whether rsync
  > is better suited than the existing implementation, with the given actual
  > arguments. I suspect, that small directories and/or short files are
  > better served with the current implementation.

Do we really need to decide this?  Wouldn't it be ok to use rsync
for all cases (where Tramp is not in use)?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : Re: Dired C idea

[Drew Adams]

>> Should be feasible. The hard part is a rule to determine, whether rsync
>> is better suited than the existing implementation, with the given actual
>> arguments. I suspect, that small directories and/or short files are
>> better served with the current implementation.
> 
> Do we really need to decide this?  Wouldn't it be ok to use rsync
> for all cases (where Tramp is not in use)?

Based on your other mail on the subject today, aren't
you really asking "Wouldn't it be preferable...?"
instead of "Wouldn't it be OK...?"

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

>   > Should be feasible. The hard part is a rule to determine, whether rsync
>   > is better suited than the existing implementation, with the given actual
>   > arguments. I suspect, that small directories and/or short files are
>   > better served with the current implementation.
>
> Do we really need to decide this?  Wouldn't it be ok to use rsync
> for all cases (where Tramp is not in use)?

I would like to see some benchmark figures first. For example, I'm not
sure that rsync performs better when the target directory does not exist
yet, and there's nothing to sync.

Best regards, Michael.

Re: Dired C idea

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>   > is better suited than the existing implementation, with the given actual
>>   > arguments. I suspect, that small directories and/or short files are
>>   > better served with the current implementation.
>>
>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>> for all cases (where Tramp is not in use)?
>
> I would like to see some benchmark figures first. For example, I'm not
> sure that rsync performs better when the target directory does not exist
> yet, and there's nothing to sync.

I made some comparison between "cp -pr" and "rsync -pr" on shell
level. I've copied the whole Emacs tree to somewhere else. First the
case, that the target does not exist:

$ time cp -pr ~/src/emacs /tmp/emacs.cp
0.086u 1.322s 0:04.43 31.6%	0+0k 1817688+0io 1pf+0w

$ time rsync -pr ~/src/emacs /tmp/emacs.rsync
5.776u 2.069s 0:05.76 135.9%	0+0k 1384384+0io 1pf+0w

"rsync -pr" is slower than "cp -pr". Now I've applied the rsync command
again, to the unchanged directory tree with exisiting target:

$ time rsync -pr ~/src/emacs /tmp/emacs.rsync
6.116u 2.457s 0:06.63 129.1%	0+0k 1069632+0io 4pf+0w

It is even a little bit slower than the first rsync run. So I don't
know, whether there is really a performance boost when we use rsync for
local directory copies.

OTOH, `copy-directory' in files.el runs recursively `copy-file',
file-wise. Perhaps we could improve this implementation on C level.

Best regards, Michael.

Re: Dired C idea

[Gregory Heytings]

>> I would like to see some benchmark figures first. For example, I'm not 
>> sure that rsync performs better when the target directory does not 
>> exist yet, and there's nothing to sync.
>
> I made some comparison between "cp -pr" and "rsync -pr" on shell level. 
> [...] "rsync -pr" is slower than "cp -pr".
>

Yes, nothing can beat cp; rsync is always slower (and uses more memory) 
compared to cp because it does more than just copying files.  It is faster 
only when copying files from/to a remote host when some files already 
exist on both sides (which is its original purpose).

But rsync is also much more powerful tool compared to cp, even when it is 
used locally: as Richard said it can resume an aborted/partial copy, it 
can synchronize two directories, it has options to include or exclude file 
patterns, it has a progress indicator, and so forth.  So as the manpage 
says, it is an "improved copy command".

RE: [External] : Re: Dired C idea

[Drew Adams]

> Yes, nothing can beat cp; rsync is always slower (and uses more memory)
> compared to cp because it does more than just copying files.  It is faster
> only when copying files from/to a remote host when some files already
> exist on both sides (which is its original purpose).
> 
> But rsync is also much more powerful tool compared to cp, even when it is
> used locally: as Richard said it can resume an aborted/partial copy, it
> can synchronize two directories, it has options to include or exclude file
> patterns, it has a progress indicator, and so forth.  So as the manpage
> says, it is an "improved copy command".

As an ignoramus in this area, this reaffirms my guess
that we should offer both, and `C' in Dired should be
kept for ordinary copy, with possibly a new Dired key
being given to rsync.

Re: [External] : Re: Dired C idea

[Pankaj Jangid]

Drew Adams <drew.adams@oracle.com> writes:

> As an ignoramus in this area, this reaffirms my guess
> that we should offer both, and `C' in Dired should be
> kept for ordinary copy, with possibly a new Dired key
> being given to rsync.

Or may be that we provide just a new function, similar to the case of
(list-buffers). Now people have a choice to re-bind ‘C-x C-b’ to
(ibuffer) family of functions.

RE: [External] : Re: Dired C idea

[Drew Adams]

 
> > As an ignoramus in this area, this reaffirms my guess
> > that we should offer both, and `C' in Dired should be
> > kept for ordinary copy, with possibly a new Dired key
> > being given to rsync.
> 
> Or may be that we provide just a new function, similar to the case of
> (list-buffers). Now people have a choice to re-bind ‘C-x C-b’ to
> (ibuffer) family of functions.

Yes, that would be my (ignorant) preference, as
I think I mentioned in the beginning.  But _if_
it's decided to give the new command a key binding
right away, I'm suggesting that it not take over
`C' from `dired-do-copy' as a default binding.

Re: Dired C idea

[Arthur Miller]

Gregory Heytings <gregory@heytings.org> writes:

>>> I would like to see some benchmark figures first. For example, I'm not sure
>>> that rsync performs better when the target directory does not exist yet, and
>>> there's nothing to sync.
>>
>> I made some comparison between "cp -pr" and "rsync -pr" on shell level. [...]
>> "rsync -pr" is slower than "cp -pr".
>>
>
> Yes, nothing can beat cp;

Is cp really bottleneck at all? Isn't it more likely IO that is limiting
operation?

> to cp because it does more than just copying files.  It is faster only when
> copying files from/to a remote host when some files already exist on both sides
> (which is its original purpose).
>
> But rsync is also much more powerful tool compared to cp, even when it is used
> locally: as Richard said it can resume an aborted/partial copy, it can
> synchronize two directories, it has options to include or exclude file patterns,
> it has a progress indicator, and so forth.  So as the manpage says, it is an
> "improved copy command".

rsync is nice, it would certainly be nice to have an built-in Emacs
interface to it, if someone has time to write it.

Re: Dired C idea

[Gregory Heytings]

>>>> I would like to see some benchmark figures first. For example, I'm 
>>>> not sure that rsync performs better when the target directory does 
>>>> not exist yet, and there's nothing to sync.
>>>
>>> I made some comparison between "cp -pr" and "rsync -pr" on shell 
>>> level. [...] "rsync -pr" is slower than "cp -pr".
>>
>> Yes, nothing can beat cp;
>
> Is cp really bottleneck at all? Isn't it more likely IO that is limiting 
> operation?
>

cp does (almost) nothing except IO, so indeed IO is the bottleneck. 
rsync does more than IO, so it is slower.

Note however that, as Fabrice just said, you should not compare "cp -pr" 
with "rsync -pr", you should compare "cp -a" with "rsync -a".  After 
"rsync -a SRC DST", another "rsync -a SRC DST" is much, much faster, and 
when DST already contains 50% of the files in SRC, "rsync -a SRC DST" will 
take ~50% of the time of a full copy.

Re: Dired C idea

[Stefan Monnier]

> Note however that, as Fabrice just said, you should not compare "cp -pr"
> with "rsync -pr", you should compare "cp -a" with "rsync -a".

Reminds me that I recently noticed my muscle memory writes "cp -a" and
I can't remember the last time I used `cp` without `-a`.


        Stefan

Re: Dired C idea

[Andreas Schwab]

On Aug 03 2021, Gregory Heytings wrote:

> you should compare "cp -a" with "rsync -a". 

Not even that.  You need at least rsync -aHAX to be on par.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Dired C idea

[Arthur Miller]

Gregory Heytings <gregory@heytings.org> writes:

>>>>> I would like to see some benchmark figures first. For example, I'm not sure
>>>>> that rsync performs better when the target directory does not exist yet,
>>>>> and there's nothing to sync.
>>>>
>>>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>>>> level. [...] "rsync -pr" is slower than "cp -pr".
>>>
>>> Yes, nothing can beat cp;
>>
>> Is cp really bottleneck at all? Isn't it more likely IO that is limiting
>> operation?
>>
>
> cp does (almost) nothing except IO, so indeed IO is the bottleneck. rsync does
> more than IO, so it is slower.

Yes, that is what I meant.

But some nice interface to rsync for other reasons than efficiency would
be nice, maybe partly incorporated in dired, for example sync dirs dwim
style or something similar.

Re: Dired C idea

[Fabrice Bauzac-Stehly]

Michael Albinus <michael.albinus@gmx.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>   > is better suited than the existing implementation, with the given actual
>>>   > arguments. I suspect, that small directories and/or short files are
>>>   > better served with the current implementation.
>>>
>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>> for all cases (where Tramp is not in use)?
>>
>> I would like to see some benchmark figures first. For example, I'm not
>> sure that rsync performs better when the target directory does not exist
>> yet, and there's nothing to sync.
>
> I made some comparison between "cp -pr" and "rsync -pr" on shell
> level. I've copied the whole Emacs tree to somewhere else. First the
> case, that the target does not exist:

Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
You'd have to add "-t" to rsync, which should make it much quicker when
the files are already there, because then rsync will trust stat(2)'s
mtime and size, skipping the actual opening and scanning of the files'
contents.  But if we don't trust this heuristics we would add
--ignore-times.  Also if we want to skip the rsync checksum scanning, and
mimic "cp" more, we can add --whole-file.  But it depends on what we
want to do?

The original use case of rsync is, of course, to synchronize things; in
particular, it will not prompt the user before overwriting a file, which
could be dangerous.  It can silently avoid overwrites with
--ignore-existing, though.  For this reason, I'm not sure it would be
wise to place it on the C key as replacement to the copy?

Another area where rsync shines is the possibility to fine-tune the
copying: --delete, rich filters on the sender and/or the receiver sides,
--copy-dest...  But then, precise control over the options is required,
so why not achieve that with the "!" key in Dired?

  ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET

If we want to bind rsync to a Dired key, I guess its use would be to
give quick access to one (or two, with the prefix key...)  sets of rsync
options corresponding to the most common needs.  We can discuss what
they could be, for me it would probably look like this:

* (1) -a --hard-links -z --one-file-system --partial --exclude=.git/
  --exclude="*~"

* Same as (1), with --delete added

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: Dired C idea

[Eric Abrahamsen]

Fabrice Bauzac-Stehly <noon@mykolab.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>>   > is better suited than the existing implementation, with the given actual
>>>>   > arguments. I suspect, that small directories and/or short files are
>>>>   > better served with the current implementation.
>>>>
>>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>>> for all cases (where Tramp is not in use)?
>>>
>>> I would like to see some benchmark figures first. For example, I'm not
>>> sure that rsync performs better when the target directory does not exist
>>> yet, and there's nothing to sync.
>>
>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>> level. I've copied the whole Emacs tree to somewhere else. First the
>> case, that the target does not exist:
>
> Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
> You'd have to add "-t" to rsync, which should make it much quicker when
> the files are already there, because then rsync will trust stat(2)'s
> mtime and size, skipping the actual opening and scanning of the files'
> contents.  But if we don't trust this heuristics we would add
> --ignore-times.  Also if we want to skip the rsync checksum scanning, and
> mimic "cp" more, we can add --whole-file.  But it depends on what we
> want to do?
>
> The original use case of rsync is, of course, to synchronize things; in
> particular, it will not prompt the user before overwriting a file, which
> could be dangerous.  It can silently avoid overwrites with
> --ignore-existing, though.  For this reason, I'm not sure it would be
> wise to place it on the C key as replacement to the copy?
>
> Another area where rsync shines is the possibility to fine-tune the
> copying: --delete, rich filters on the sender and/or the receiver sides,
> --copy-dest...  But then, precise control over the options is required,
> so why not achieve that with the "!" key in Dired?
>
>   ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET

Looks like a job for the newly merged transient library!

Re: Dired C idea

[Arthur Miller]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Fabrice Bauzac-Stehly <noon@mykolab.com> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>> Michael Albinus <michael.albinus@gmx.de> writes:
>>>
>>>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>>>   > is better suited than the existing implementation, with the given actual
>>>>>   > arguments. I suspect, that small directories and/or short files are
>>>>>   > better served with the current implementation.
>>>>>
>>>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>>>> for all cases (where Tramp is not in use)?
>>>>
>>>> I would like to see some benchmark figures first. For example, I'm not
>>>> sure that rsync performs better when the target directory does not exist
>>>> yet, and there's nothing to sync.
>>>
>>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>>> level. I've copied the whole Emacs tree to somewhere else. First the
>>> case, that the target does not exist:
>>
>> Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
>> You'd have to add "-t" to rsync, which should make it much quicker when
>> the files are already there, because then rsync will trust stat(2)'s
>> mtime and size, skipping the actual opening and scanning of the files'
>> contents.  But if we don't trust this heuristics we would add
>> --ignore-times.  Also if we want to skip the rsync checksum scanning, and
>> mimic "cp" more, we can add --whole-file.  But it depends on what we
>> want to do?
>>
>> The original use case of rsync is, of course, to synchronize things; in
>> particular, it will not prompt the user before overwriting a file, which
>> could be dangerous.  It can silently avoid overwrites with
>> --ignore-existing, though.  For this reason, I'm not sure it would be
>> wise to place it on the C key as replacement to the copy?
>>
>> Another area where rsync shines is the possibility to fine-tune the
>> copying: --delete, rich filters on the sender and/or the receiver sides,
>> --copy-dest...  But then, precise control over the options is required,
>> so why not achieve that with the "!" key in Dired?
>>
>>   ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET
>
> Looks like a job for the newly merged transient library!

I woujld like to see something like `sync-directories-dwim
style'. For the filters maybe something like `copy-if', `delete-if' or
some other more integrated way with Emacs and Dired. Nothing against
transient, just the way I would prefer to have it. I use rsync from
Emacs already as from the links I posted.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is even a little bit slower than the first rsync run. So I don't
  > know, whether there is really a performance boost when we use rsync for
  > local directory copies.

If the overall operation is pretty fast, it may not matter to make it
a little faster or a little slower.  Either way, it is not painful.
Using rsync in this case would be no significant benefit but also no
significant loss.

Copying large directories between disks can take many minutes.
rsync makes a big improvement in such cases.  If you copy with cp and
it dies, you've been screwed.  With rsync, you just continue it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> > I would like to see some benchmark figures first. For example, I'm not
> > sure that rsync performs better when the target directory does not exist
> > yet, and there's nothing to sync.
> 
> I made some comparison between "cp -pr" and "rsync -pr" on shell
> level.

Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.

I expect that rsync will gain equivalent capabilities in time, but it doesn’t have them yet.

-- 
Peter Oliver

Re: Dired C idea

[Eli Zaretskii]

> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> 
> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.

What's to prevent Emacs from doing the same?

Re: Dired C idea

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On Sat, 7 Aug 2021, Eli Zaretskii wrote:

>> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
>> From: Peter Oliver <p.d.oliver@mavit.org.uk>
>>
>> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.
>
> What's to prevent Emacs from doing the same?

It turns out that we already do (commit 9b21d9f9).

-- 
Peter Oliver

Re: Dired C idea

[Eli Zaretskii]

> Date: Sat, 7 Aug 2021 19:54:39 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> >> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
> >> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> >>
> >> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.
> >
> > What's to prevent Emacs from doing the same?
> 
> It turns out that we already do (commit 9b21d9f9).

Yes, I know.