Dired C idea

Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What would people think about making the C command in Dired
run rsync to do the copying of a directory>

The idea is that interrupting the copying would let you
restart it without starting from zero.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Alexandre Garreau]

Le samedi 31 juillet 2021, 03:34:12 CEST Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> What would people think about making the C command in Dired
> run rsync to do the copying of a directory>
> 
> The idea is that interrupting the copying would let you
> restart it without starting from zero.

sounds really cool and useful (especially as it would just as well allow 
to seemlessly copy to remote), but makes wonder what is cp for nowadays we 
have rsync?

just in case cp still has something not in rsync, shouldn’t we ask the 
coreutils people (current maintainer?) what could it be, so we’d know as 
well it would be neat to integrate it/them into rsync so the decision of 
switching from cp to rsync would be purely positive in any respect (and 
maybe also other places as well)

Re: Dired C idea

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

Sounds like something worth evaluating, but wonder if....

Is rsync available on all supported platforms?

Is rsync installed by default on supported platforms or will this mean that a
core operation i.e. dired, will require installation of rsync before Emacs will
work correctly?

Are there any issues needing to be considered wrt rsync and different file
systems i.e. btrfs, zfs, etc.

Should we replace cp with rsync or should we add rsync as an option/additional
command? Not sure what the impact of rsync might be with simpler cp operations.

Regards,

Tim

--
*Tim Cross*

/For gor sake stop laughing, this is serious!/

Re: Dired C idea

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Sat, 31 Jul 2021 13:35:39 +1000
> 
> Is rsync available on all supported platforms?

We can test that before invoking the command.  We do that in many
other cases.  So this issue is not serious.

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> What would people think about making the C command in Dired
> run rsync to do the copying of a directory>
>
> The idea is that interrupting the copying would let you
> restart it without starting from zero.

Internally, `copy-directory' is called, which is aware of file name
handlers. Using rsync instead of copy shouldn't break this mechanism.

If a remote directory is involved, either source or target, it is
already possible to use Tramp's `rsync' method.

Best regards, Michael.

Re: Dired C idea

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Date: Sat, 31 Jul 2021 13:35:39 +1000
>> 
>> Is rsync available on all supported platforms?
>
> We can test that before invoking the command.  We do that in many
> other cases.  So this issue is not serious.

It is already possible to run rsync form emacs, by simply calling it
from lisp code. I do it all the time :)

Why do you need it in hardcoded in C code? What extra does it offer?

There are like different versions of ellisp already, most of them spun
from this blog post:

https://truongtx.me/tmtxt-dired-async.html

https://github.com/stsquad/dired-rsync
https://oremacs.com/2016/02/24/dired-rsync/
https://gist.github.com/l3msh0/5006645

Re: Dired C idea

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 31 Jul 2021 18:03:35 +0200
> 
> Why do you need it in hardcoded in C code?

This is a misunderstanding, I think: Richard said "C command" meaning
the command bound to "C" in Dired mode.  He didn't mean to hardcode
this in C code.

Re: Dired C idea

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
>> Date: Sat, 31 Jul 2021 18:03:35 +0200
>> 
>> Why do you need it in hardcoded in C code?
>
> This is a misunderstanding, I think: Richard said "C command" meaning
> the command bound to "C" in Dired mode.  He didn't mean to hardcode
> this in C code.

Ah, sorry, indeed it is misundestanding on my part. I appologize.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Internally, `copy-directory' is called, which is aware of file name
  > handlers. Using rsync instead of copy shouldn't break this mechanism.

Should `copy-directory' itself, on local directories, run rsync?
Maybe that would be a simple way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > Internally, `copy-directory' is called, which is aware of file name
>   > handlers. Using rsync instead of copy shouldn't break this mechanism.
>
> Should `copy-directory' itself, on local directories, run rsync?
> Maybe that would be a simple way.

Should be feasible. The hard part is a rule to determine, whether rsync
is better suited than the existing implementation, with the given actual
arguments. I suspect, that small directories and/or short files are
better served with the current implementation.

There exists also the librsync library, released under GNU LGPL v2.1. It
might help to implement rsync in Emacs core, instead of calling an
external process.

Best regards, Michael.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Should be feasible. The hard part is a rule to determine, whether rsync
  > is better suited than the existing implementation, with the given actual
  > arguments. I suspect, that small directories and/or short files are
  > better served with the current implementation.

Do we really need to decide this?  Wouldn't it be ok to use rsync
for all cases (where Tramp is not in use)?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

>   > Should be feasible. The hard part is a rule to determine, whether rsync
>   > is better suited than the existing implementation, with the given actual
>   > arguments. I suspect, that small directories and/or short files are
>   > better served with the current implementation.
>
> Do we really need to decide this?  Wouldn't it be ok to use rsync
> for all cases (where Tramp is not in use)?

I would like to see some benchmark figures first. For example, I'm not
sure that rsync performs better when the target directory does not exist
yet, and there's nothing to sync.

Best regards, Michael.

Re: Dired C idea

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>   > is better suited than the existing implementation, with the given actual
>>   > arguments. I suspect, that small directories and/or short files are
>>   > better served with the current implementation.
>>
>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>> for all cases (where Tramp is not in use)?
>
> I would like to see some benchmark figures first. For example, I'm not
> sure that rsync performs better when the target directory does not exist
> yet, and there's nothing to sync.

I made some comparison between "cp -pr" and "rsync -pr" on shell
level. I've copied the whole Emacs tree to somewhere else. First the
case, that the target does not exist:

$ time cp -pr ~/src/emacs /tmp/emacs.cp
0.086u 1.322s 0:04.43 31.6%	0+0k 1817688+0io 1pf+0w

$ time rsync -pr ~/src/emacs /tmp/emacs.rsync
5.776u 2.069s 0:05.76 135.9%	0+0k 1384384+0io 1pf+0w

"rsync -pr" is slower than "cp -pr". Now I've applied the rsync command
again, to the unchanged directory tree with exisiting target:

$ time rsync -pr ~/src/emacs /tmp/emacs.rsync
6.116u 2.457s 0:06.63 129.1%	0+0k 1069632+0io 4pf+0w

It is even a little bit slower than the first rsync run. So I don't
know, whether there is really a performance boost when we use rsync for
local directory copies.

OTOH, `copy-directory' in files.el runs recursively `copy-file',
file-wise. Perhaps we could improve this implementation on C level.

Best regards, Michael.

Re: Dired C idea

[Gregory Heytings]

>> I would like to see some benchmark figures first. For example, I'm not 
>> sure that rsync performs better when the target directory does not 
>> exist yet, and there's nothing to sync.
>
> I made some comparison between "cp -pr" and "rsync -pr" on shell level. 
> [...] "rsync -pr" is slower than "cp -pr".
>

Yes, nothing can beat cp; rsync is always slower (and uses more memory) 
compared to cp because it does more than just copying files.  It is faster 
only when copying files from/to a remote host when some files already 
exist on both sides (which is its original purpose).

But rsync is also much more powerful tool compared to cp, even when it is 
used locally: as Richard said it can resume an aborted/partial copy, it 
can synchronize two directories, it has options to include or exclude file 
patterns, it has a progress indicator, and so forth.  So as the manpage 
says, it is an "improved copy command".

Re: Dired C idea

[Fabrice Bauzac-Stehly]

Michael Albinus <michael.albinus@gmx.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>   > is better suited than the existing implementation, with the given actual
>>>   > arguments. I suspect, that small directories and/or short files are
>>>   > better served with the current implementation.
>>>
>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>> for all cases (where Tramp is not in use)?
>>
>> I would like to see some benchmark figures first. For example, I'm not
>> sure that rsync performs better when the target directory does not exist
>> yet, and there's nothing to sync.
>
> I made some comparison between "cp -pr" and "rsync -pr" on shell
> level. I've copied the whole Emacs tree to somewhere else. First the
> case, that the target does not exist:

Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
You'd have to add "-t" to rsync, which should make it much quicker when
the files are already there, because then rsync will trust stat(2)'s
mtime and size, skipping the actual opening and scanning of the files'
contents.  But if we don't trust this heuristics we would add
--ignore-times.  Also if we want to skip the rsync checksum scanning, and
mimic "cp" more, we can add --whole-file.  But it depends on what we
want to do?

The original use case of rsync is, of course, to synchronize things; in
particular, it will not prompt the user before overwriting a file, which
could be dangerous.  It can silently avoid overwrites with
--ignore-existing, though.  For this reason, I'm not sure it would be
wise to place it on the C key as replacement to the copy?

Another area where rsync shines is the possibility to fine-tune the
copying: --delete, rich filters on the sender and/or the receiver sides,
--copy-dest...  But then, precise control over the options is required,
so why not achieve that with the "!" key in Dired?

  ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET

If we want to bind rsync to a Dired key, I guess its use would be to
give quick access to one (or two, with the prefix key...)  sets of rsync
options corresponding to the most common needs.  We can discuss what
they could be, for me it would probably look like this:

* (1) -a --hard-links -z --one-file-system --partial --exclude=.git/
  --exclude="*~"

* Same as (1), with --delete added

-- 
Fabrice Bauzac-Stehly
PGP 01EEACF8244E9C14B551C5256ADA5F189BD322B6

Re: Dired C idea

[Arthur Miller]

Gregory Heytings <gregory@heytings.org> writes:

>>> I would like to see some benchmark figures first. For example, I'm not sure
>>> that rsync performs better when the target directory does not exist yet, and
>>> there's nothing to sync.
>>
>> I made some comparison between "cp -pr" and "rsync -pr" on shell level. [...]
>> "rsync -pr" is slower than "cp -pr".
>>
>
> Yes, nothing can beat cp;

Is cp really bottleneck at all? Isn't it more likely IO that is limiting
operation?

> to cp because it does more than just copying files.  It is faster only when
> copying files from/to a remote host when some files already exist on both sides
> (which is its original purpose).
>
> But rsync is also much more powerful tool compared to cp, even when it is used
> locally: as Richard said it can resume an aborted/partial copy, it can
> synchronize two directories, it has options to include or exclude file patterns,
> it has a progress indicator, and so forth.  So as the manpage says, it is an
> "improved copy command".

rsync is nice, it would certainly be nice to have an built-in Emacs
interface to it, if someone has time to write it.

Re: Dired C idea

[Eric Abrahamsen]

Fabrice Bauzac-Stehly <noon@mykolab.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>>   > is better suited than the existing implementation, with the given actual
>>>>   > arguments. I suspect, that small directories and/or short files are
>>>>   > better served with the current implementation.
>>>>
>>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>>> for all cases (where Tramp is not in use)?
>>>
>>> I would like to see some benchmark figures first. For example, I'm not
>>> sure that rsync performs better when the target directory does not exist
>>> yet, and there's nothing to sync.
>>
>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>> level. I've copied the whole Emacs tree to somewhere else. First the
>> case, that the target does not exist:
>
> Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
> You'd have to add "-t" to rsync, which should make it much quicker when
> the files are already there, because then rsync will trust stat(2)'s
> mtime and size, skipping the actual opening and scanning of the files'
> contents.  But if we don't trust this heuristics we would add
> --ignore-times.  Also if we want to skip the rsync checksum scanning, and
> mimic "cp" more, we can add --whole-file.  But it depends on what we
> want to do?
>
> The original use case of rsync is, of course, to synchronize things; in
> particular, it will not prompt the user before overwriting a file, which
> could be dangerous.  It can silently avoid overwrites with
> --ignore-existing, though.  For this reason, I'm not sure it would be
> wise to place it on the C key as replacement to the copy?
>
> Another area where rsync shines is the possibility to fine-tune the
> copying: --delete, rich filters on the sender and/or the receiver sides,
> --copy-dest...  But then, precise control over the options is required,
> so why not achieve that with the "!" key in Dired?
>
>   ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET

Looks like a job for the newly merged transient library!

Re: Dired C idea

[Gregory Heytings]

>>>> I would like to see some benchmark figures first. For example, I'm 
>>>> not sure that rsync performs better when the target directory does 
>>>> not exist yet, and there's nothing to sync.
>>>
>>> I made some comparison between "cp -pr" and "rsync -pr" on shell 
>>> level. [...] "rsync -pr" is slower than "cp -pr".
>>
>> Yes, nothing can beat cp;
>
> Is cp really bottleneck at all? Isn't it more likely IO that is limiting 
> operation?
>

cp does (almost) nothing except IO, so indeed IO is the bottleneck. 
rsync does more than IO, so it is slower.

Note however that, as Fabrice just said, you should not compare "cp -pr" 
with "rsync -pr", you should compare "cp -a" with "rsync -a".  After 
"rsync -a SRC DST", another "rsync -a SRC DST" is much, much faster, and 
when DST already contains 50% of the files in SRC, "rsync -a SRC DST" will 
take ~50% of the time of a full copy.

Re: Dired C idea

[Stefan Monnier]

> Note however that, as Fabrice just said, you should not compare "cp -pr"
> with "rsync -pr", you should compare "cp -a" with "rsync -a".

Reminds me that I recently noticed my muscle memory writes "cp -a" and
I can't remember the last time I used `cp` without `-a`.


        Stefan

Re: Dired C idea

[Andreas Schwab]

On Aug 03 2021, Gregory Heytings wrote:

> you should compare "cp -a" with "rsync -a". 

Not even that.  You need at least rsync -aHAX to be on par.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Dired C idea

[Arthur Miller]

Gregory Heytings <gregory@heytings.org> writes:

>>>>> I would like to see some benchmark figures first. For example, I'm not sure
>>>>> that rsync performs better when the target directory does not exist yet,
>>>>> and there's nothing to sync.
>>>>
>>>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>>>> level. [...] "rsync -pr" is slower than "cp -pr".
>>>
>>> Yes, nothing can beat cp;
>>
>> Is cp really bottleneck at all? Isn't it more likely IO that is limiting
>> operation?
>>
>
> cp does (almost) nothing except IO, so indeed IO is the bottleneck. rsync does
> more than IO, so it is slower.

Yes, that is what I meant.

But some nice interface to rsync for other reasons than efficiency would
be nice, maybe partly incorporated in dired, for example sync dirs dwim
style or something similar.

Re: Dired C idea

[Arthur Miller]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Fabrice Bauzac-Stehly <noon@mykolab.com> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>> Michael Albinus <michael.albinus@gmx.de> writes:
>>>
>>>>>   > Should be feasible. The hard part is a rule to determine, whether rsync
>>>>>   > is better suited than the existing implementation, with the given actual
>>>>>   > arguments. I suspect, that small directories and/or short files are
>>>>>   > better served with the current implementation.
>>>>>
>>>>> Do we really need to decide this?  Wouldn't it be ok to use rsync
>>>>> for all cases (where Tramp is not in use)?
>>>>
>>>> I would like to see some benchmark figures first. For example, I'm not
>>>> sure that rsync performs better when the target directory does not exist
>>>> yet, and there's nothing to sync.
>>>
>>> I made some comparison between "cp -pr" and "rsync -pr" on shell
>>> level. I've copied the whole Emacs tree to somewhere else. First the
>>> case, that the target does not exist:
>>
>> Beware, "cp -pr" preserves timestamps, while "rsync -pr" does not.
>> You'd have to add "-t" to rsync, which should make it much quicker when
>> the files are already there, because then rsync will trust stat(2)'s
>> mtime and size, skipping the actual opening and scanning of the files'
>> contents.  But if we don't trust this heuristics we would add
>> --ignore-times.  Also if we want to skip the rsync checksum scanning, and
>> mimic "cp" more, we can add --whole-file.  But it depends on what we
>> want to do?
>>
>> The original use case of rsync is, of course, to synchronize things; in
>> particular, it will not prompt the user before overwriting a file, which
>> could be dangerous.  It can silently avoid overwrites with
>> --ignore-existing, though.  For this reason, I'm not sure it would be
>> wise to place it on the C key as replacement to the copy?
>>
>> Another area where rsync shines is the possibility to fine-tune the
>> copying: --delete, rich filters on the sender and/or the receiver sides,
>> --copy-dest...  But then, precise control over the options is required,
>> so why not achieve that with the "!" key in Dired?
>>
>>   ! rsync -rt "--filter=-s *.dat" --delete * /tmp/ RET
>
> Looks like a job for the newly merged transient library!

I woujld like to see something like `sync-directories-dwim
style'. For the filters maybe something like `copy-if', `delete-if' or
some other more integrated way with Emacs and Dired. Nothing against
transient, just the way I would prefer to have it. I use rsync from
Emacs already as from the links I posted.

Re: Dired C idea

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is even a little bit slower than the first rsync run. So I don't
  > know, whether there is really a performance boost when we use rsync for
  > local directory copies.

If the overall operation is pretty fast, it may not matter to make it
a little faster or a little slower.  Either way, it is not painful.
Using rsync in this case would be no significant benefit but also no
significant loss.

Copying large directories between disks can take many minutes.
rsync makes a big improvement in such cases.  If you copy with cp and
it dies, you've been screwed.  With rsync, you just continue it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Dired C idea

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> > I would like to see some benchmark figures first. For example, I'm not
> > sure that rsync performs better when the target directory does not exist
> > yet, and there's nothing to sync.
> 
> I made some comparison between "cp -pr" and "rsync -pr" on shell
> level.

Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.

I expect that rsync will gain equivalent capabilities in time, but it doesn’t have them yet.

-- 
Peter Oliver

Re: Dired C idea

[Eli Zaretskii]

> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> 
> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.

What's to prevent Emacs from doing the same?

Re: Dired C idea

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On Sat, 7 Aug 2021, Eli Zaretskii wrote:

>> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
>> From: Peter Oliver <p.d.oliver@mavit.org.uk>
>>
>> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.
>
> What's to prevent Emacs from doing the same?

It turns out that we already do (commit 9b21d9f9).

-- 
Peter Oliver

Re: Dired C idea

[Eli Zaretskii]

> Date: Sat, 7 Aug 2021 19:54:39 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> >> Date: Sat, 7 Aug 2021 19:22:02 +0100 (BST)
> >> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> >>
> >> Something else to consider is that from the next major version of GNU Coreutils, cp will default to copy-on-write (“--reflink=auto”) on supported filesystems.  This is lightning fast, and saves disk space too.
> >
> > What's to prevent Emacs from doing the same?
> 
> It turns out that we already do (commit 9b21d9f9).

Yes, I know.