From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Tomas Volf <~@wolfsden.cz> Newsgroups: gmane.lisp.guile.devel Subject: [PATCH] Add copy-on-write support to scm_copy_file. Date: Wed, 29 Nov 2023 17:17:39 +0100 Message-ID: <20231129162027.14422-1-~@wolfsden.cz> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="442"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Tomas Volf <~@wolfsden.cz> To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Wed Nov 29 17:21:30 2023 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1r8NJY-000AQw-Dr for guile-devel@m.gmane-mx.org; Wed, 29 Nov 2023 17:21:28 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r8NIy-0006ki-VX; Wed, 29 Nov 2023 11:20:55 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r8NIk-0006jI-If for guile-devel@gnu.org; Wed, 29 Nov 2023 11:20:40 -0500 Original-Received: from wolfsden.cz ([37.205.8.62]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r8NIh-0007Oe-T7 for guile-devel@gnu.org; Wed, 29 Nov 2023 11:20:38 -0500 Original-Received: by wolfsden.cz (Postfix, from userid 104) id 14AEE2608B2; Wed, 29 Nov 2023 16:20:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1701274831; bh=QW99eb6eu+0pXP+JWDL9V/wZxGKxMl3EwavcsxWaAmg=; h=From:To:Cc:Subject:Date; b=KZkZf+fEz7H9VJ8iC8EDukK3S3JDIZvJFfIs69JKzyGt/dddfxxpvI2B/gMpsNY4p Qx2IdK/OpXFbQNnXRDW6ahH5PQ8y7k3UMlStcvlVyGyWFVLSG/pgbPxOR7/WZJE7x6 AyTI46/uP58KcCEMWhhAZKf6YB/EyPocebHC54PWJrlnjJFVEWbkVN+mqEV8/mMIQh GySZoJodMJ9mwhIW4RFuZdjTZQR/EhhPltlvkLbcP9kagE8mWpSn4KnE1kH+6fFzT5 AKtbMbn7sDWCGlUszzf88rj9kh3ci3Ifpg+NMfOZ6dO44ZyMEU3ZQGWuQEr4lfLYIX 8JN2hTJimNuHkHMg11MB550gPco426YihF/YT6V2IjCk6mT0XYPz4U6hoLOekVlo+P 5FpjBNIdcCYTqFN9tgLkxIs9uGmpCYYJY0BJXazmv15EzwWZyn+YRPRgHFJUK69ItE B0AqAMe6AamsfolUSeHY8sMU/xuISeIY3FchIRgw8FW6TdWC7ICUDRgkddRwRKViL8 V0p8BTtEzq/IwGzRcZAMEbUXOIaUX2K6hEBswoKy918JvIktaGcu1OavDtSqiSCowz KKBPm5jY8G0OjaSkbVZHlAKFgbATNN1xuSbEM3yvOUKnUnjmlJuGBvqdwqWmhjwLYI UqLrrLCVVdNcqvyDs5fZPm5M= Original-Received: from localhost (unknown [81.17.16.69]) by wolfsden.cz (Postfix) with ESMTPSA id 24E5125F46F; Wed, 29 Nov 2023 16:20:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1701274830; bh=QW99eb6eu+0pXP+JWDL9V/wZxGKxMl3EwavcsxWaAmg=; h=From:To:Cc:Subject:Date; b=sZ1wt5dCm3oKVSi4Sm2KfHWM2Z3POWNY/sEMlq95WNFvQxmbBywzU+2fyI83i41Qk yu71AeDHgKB6t6sxMw2YvELtPKrR42lbJH7Tv136Akt1VpVTZ5yW0IZaauwuYsHQAW 15W/vBEyxy+gJGcDDhZ+BO/nXHKqgegQWDzkFJ7qThMMcl9MG9KVZldP/nSJVpx7DU TEoy5zmxVY1tdqXQiwk6em5xFND14o0HotPXialquStK12pw7WsTVOS8nqqx9TlqjJ ErlnOGNiUbd7RZWaOyp1qegNFoxrx2r1E5UbmZK1SDUFej6FTtks/QY+4CRYL4a9XI 6nM5myqHAkeBV67OFNrVKx9NzPgc3aOs8ui9BRz+/4uM312QYniBfvp4m7hSSXmgct W60VyAEH/nQyo1sR67W6MsXUTB4kzhtSPOp9I1ddKpXdMx/MjuMUytItku1+SugxLw QzLef9hqF3G+zTC0UXvcAMHbNpo/GhvktKr5iePGCpE3jeI7Q9+jprd9Yue9AG10tX LrQzLGV7y6JIwNkmZSdlLJd/0rIFqNkoNGPRqonYhW4K7rwPi00prcjfC1SC7aFPc/ DDDa0Y46mCrFkOAaprmyyxz/E6ctIeUlFRDgcwvgts9wAAdycSga43mYheaHt1+sS3 IgjKipb90S7VLf8cfyFSbtf0= Original-Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 57e1c13b; Wed, 29 Nov 2023 16:20:29 +0000 (UTC) X-Mailer: git-send-email 2.41.0 Received-SPF: none client-ip=37.205.8.62; envelope-from=ws@wolfsnet.cz; helo=wolfsden.cz X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, UNPARSEABLE_RELAY=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.devel:22141 Archived-At: On modern file-systems (BTRFS, ZFS) it is possible to copy a file using copy-on-write method. For large files it has the advantage of being much faster and saving disk space (since identical extents are not duplicated). This feature is stable and for example coreutils' `cp' does use it automatically (see --reflink). This commit adds support for this feature into our copy-file (scm_copy_file) procedure. Same as `cp', it defaults to 'auto, meaning the copy-on-write is attempted, and in case of failure the regular copy is performed. No tests are provided, because the behavior depends on the system, underlying file-system and its configuration. That makes it challenging to write a test for it. Manual testing was performed instead: $ btrfs filesystem du /tmp/cow* Total Exclusive Set shared Filename 36.00KiB 36.00KiB 0.00B /tmp/cow $ cat cow-test.scm (copy-file "/tmp/cow" "/tmp/cow-unspecified") (copy-file "/tmp/cow" "/tmp/cow-always" #:copy-on-write 'always) (copy-file "/tmp/cow" "/tmp/cow-auto" #:copy-on-write 'auto) (copy-file "/tmp/cow" "/tmp/cow-never" #:copy-on-write 'never) (copy-file "/tmp/cow" "/dev/shm/cow-unspecified") (copy-file "/tmp/cow" "/dev/shm/cow-auto" #:copy-on-write 'auto) (copy-file "/tmp/cow" "/dev/shm/cow-never" #:copy-on-write 'never) $ ./meta/guile -s cow-test.scm $ btrfs filesystem du /tmp/cow* Total Exclusive Set shared Filename 36.00KiB 0.00B 36.00KiB /tmp/cow 36.00KiB 0.00B 36.00KiB /tmp/cow-always 36.00KiB 0.00B 36.00KiB /tmp/cow-auto 36.00KiB 36.00KiB 0.00B /tmp/cow-never 36.00KiB 0.00B 36.00KiB /tmp/cow-unspecified $ sha1sum /tmp/cow* /dev/shm/cow* 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /tmp/cow 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /tmp/cow-always 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /tmp/cow-auto 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /tmp/cow-never 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /tmp/cow-unspecified 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /dev/shm/cow-auto 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /dev/shm/cow-never 4c665f87b5dc2e7d26279c4b48968d085e1ace32 /dev/shm/cow-unspecified This commit also adds to new failure modes for (copy-file). Failure to copy-on-write when 'always was passed in: scheme@(guile-user)> (copy-file "/tmp/cow" "/dev/shm/cow" #:copy-on-write 'always) ice-9/boot-9.scm:1676:22: In procedure raise-exception: In procedure copy-file: copy-on-write failed: Invalid cross-device link Passing in invalid value for the #:copy-on-write keyword argument: scheme@(guile-user)> (copy-file "/tmp/cow" "/dev/shm/cow" #:copy-on-write 'nevr) ice-9/boot-9.scm:1676:22: In procedure raise-exception: In procedure copy-file: invalid value for #:copy-on-write: nevr * NEWS: Add note for copy-file supporting copy-on-write. * configure.ac: Check for linux/fs.h. * doc/ref/posix.texi (File System)[copy-file]: Document the new signature. * libguile/filesys.c (clone_file): New function cloning a file using FICLONE, if supported. (k_copy_on_write): New keyword. (sym_always, sym_auto, sym_never): New symbols. (scm_copy_file): New #:copy-on-write keyword argument. Attempt copy-on-write copy by default. * libguile/filesys.h: Update signature for scm_copy_file. --- My first C patch to Guile. I am not used to the GNU coding style, so I apologize in advance if I got something wrong. I tried. I believe the 'auto is correct default, mimicking how coreutils' cp works. NEWS | 9 ++++++ configure.ac | 1 + doc/ref/posix.texi | 9 +++++- libguile/filesys.c | 74 +++++++++++++++++++++++++++++++++++++++------- libguile/filesys.h | 2 +- 5 files changed, 82 insertions(+), 13 deletions(-) diff --git a/NEWS b/NEWS index b319404d7..9147098c9 100644 --- a/NEWS +++ b/NEWS @@ -21,6 +21,15 @@ definitely unused---this is notably the case for modules that are only used at macro-expansion time, such as (srfi srfi-26). In those cases, the compiler reports it as "possibly unused". +** copy-file now supports copy-on-write + +The copy-file procedure now takes an additional keyword argument, +#:copy-on-write, specifying whether copy-on-write should be done, if the +underlying file-system supports it. Possible values are 'always, 'auto +and 'never, with 'auto being the default. + +This speeds up copying large files a lot while saving the disk space. + * Bug fixes ** (ice-9 suspendable-ports) incorrect UTF-8 decoding diff --git a/configure.ac b/configure.ac index d0a2dc79b..c46586e9b 100644 --- a/configure.ac +++ b/configure.ac @@ -418,6 +418,7 @@ AC_SUBST([SCM_I_GSC_HAVE_STRUCT_DIRENT64]) # sys/sendfile.h - non-POSIX, found in glibc # AC_CHECK_HEADERS([complex.h fenv.h io.h memory.h process.h \ +linux/fs.h \ sys/dir.h sys/ioctl.h sys/select.h \ sys/time.h sys/timeb.h sys/times.h sys/stdtypes.h sys/types.h \ sys/utime.h unistd.h utime.h pwd.h grp.h sys/utsname.h \ diff --git a/doc/ref/posix.texi b/doc/ref/posix.texi index fec42d061..d26808d91 100644 --- a/doc/ref/posix.texi +++ b/doc/ref/posix.texi @@ -896,10 +896,17 @@ of @code{delete-file}. Why doesn't POSIX have a @code{rmdirat} function for this instead? No idea! @end deffn -@deffn {Scheme Procedure} copy-file oldfile newfile +@deffn {Scheme Procedure} copy-file @var{oldfile} @var{newfile} @ + [#:copy-on-write='auto] @deffnx {C Function} scm_copy_file (oldfile, newfile) Copy the file specified by @var{oldfile} to @var{newfile}. The return value is unspecified. + +@code{#:copy-on-write} keyword argument determines whether copy-on-write +copy should be attempted and the behavior in case of failure. Possible +values are @code{'always} (attempt the copy-on-write, return error if it +fails), @code{'auto} (attempt the copy-on-write, fallback to regular +copy if it fails) and @code{'never} (perform the regular copy). @end deffn @deffn {Scheme Procedure} sendfile out in count [offset] diff --git a/libguile/filesys.c b/libguile/filesys.c index 1f0bba556..4fb8b9831 100644 --- a/libguile/filesys.c +++ b/libguile/filesys.c @@ -67,6 +67,11 @@ # include #endif +#if defined(HAVE_SYS_IOCTL_H) && defined(HAVE_LINUX_FS_H) +# include +# include +#endif + #include "async.h" #include "boolean.h" #include "dynwind.h" @@ -75,6 +80,7 @@ #include "fports.h" #include "gsubr.h" #include "iselect.h" +#include "keywords.h" #include "list.h" #include "load.h" /* for scm_i_mirror_backslashes */ #include "modules.h" @@ -1255,20 +1261,49 @@ SCM_DEFINE (scm_readlink, "readlink", 1, 0, 0, } #undef FUNC_NAME -SCM_DEFINE (scm_copy_file, "copy-file", 2, 0, 0, - (SCM oldfile, SCM newfile), +static int +clone_file (int oldfd, int newfd) +{ +#ifdef FICLONE + return ioctl (newfd, FICLONE, oldfd); +#else + (void)oldfd; + (void)newfd; + errno = EOPNOTSUPP; + return -1; +#endif +} + +SCM_KEYWORD (k_copy_on_write, "copy-on-write"); +SCM_SYMBOL (sym_always, "always"); +SCM_SYMBOL (sym_auto, "auto"); +SCM_SYMBOL (sym_never, "never"); + +SCM_DEFINE (scm_copy_file, "copy-file", 2, 0, 1, + (SCM oldfile, SCM newfile, SCM rest), "Copy the file specified by @var{oldfile} to @var{newfile}.\n" - "The return value is unspecified.") + "The return value is unspecified.\n" + "\n" + "@code{#:copy-on-write} keyword argument determines whether " + "copy-on-write copy should be attempted and the " + "behavior in case of failure. Possible values are " + "@code{'always} (attempt the copy-on-write, return error if " + "it fails), @code{'auto} (attempt the copy-on-write, " + "fallback to regular copy if it fails) and @code{'never} " + "(perform the regular copy)." + ) #define FUNC_NAME s_scm_copy_file { char *c_oldfile, *c_newfile; int oldfd, newfd; int n, rv; + SCM cow = sym_auto; + int clone_res; char buf[BUFSIZ]; struct stat_or_stat64 oldstat; scm_dynwind_begin (0); - + c_oldfile = scm_to_locale_string (oldfile); scm_dynwind_free (c_oldfile); c_newfile = scm_to_locale_string (newfile); @@ -1292,13 +1327,30 @@ SCM_DEFINE (scm_copy_file, "copy-file", 2, 0, 0, SCM_SYSERROR; } - while ((n = read (oldfd, buf, sizeof buf)) > 0) - if (write (newfd, buf, n) != n) - { - close (oldfd); - close (newfd); - SCM_SYSERROR; - } + scm_c_bind_keyword_arguments ("copy-file", rest, 0, + k_copy_on_write, &cow, + SCM_UNDEFINED); + + if (scm_is_eq (cow, sym_always) || scm_is_eq (cow, sym_auto)) + clone_res = clone_file(oldfd, newfd); + else if (scm_is_eq (cow, sym_never)) + clone_res = -1; + else + scm_misc_error ("copy-file", + "invalid value for #:copy-on-write: ~S", + scm_list_1 (cow)); + + if (scm_is_eq (cow, sym_always) && clone_res) + scm_syserror ("copy-file: copy-on-write failed"); + + if (clone_res) + while ((n = read (oldfd, buf, sizeof buf)) > 0) + if (write (newfd, buf, n) != n) + { + close (oldfd); + close (newfd); + SCM_SYSERROR; + } close (oldfd); if (close (newfd) == -1) SCM_SYSERROR; diff --git a/libguile/filesys.h b/libguile/filesys.h index 1ce50d30e..4f620dfef 100644 --- a/libguile/filesys.h +++ b/libguile/filesys.h @@ -73,7 +73,7 @@ SCM_API SCM scm_symlink (SCM oldpath, SCM newpath); SCM_API SCM scm_symlinkat (SCM dir, SCM oldpath, SCM newpath); SCM_API SCM scm_readlink (SCM path); SCM_API SCM scm_lstat (SCM str); -SCM_API SCM scm_copy_file (SCM oldfile, SCM newfile); +SCM_API SCM scm_copy_file (SCM oldfile, SCM newfile, SCM rest); SCM_API SCM scm_mkstemp (SCM tmpl); SCM_API SCM scm_mkdtemp (SCM tmpl); SCM_API SCM scm_dirname (SCM filename); -- 2.41.0