From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id IIFeI4QwkGEnZQAAgWs5BA (envelope-from ) for ; Sat, 13 Nov 2021 22:39:16 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id wH/nHoQwkGFAKAAAbx9fmQ (envelope-from ) for ; Sat, 13 Nov 2021 21:39:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4629532A82 for ; Sat, 13 Nov 2021 22:39:15 +0100 (CET) Received: from localhost ([::1]:51336 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mm0jy-0003WS-AB for larch@yhetil.org; Sat, 13 Nov 2021 16:39:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46362) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mm0jm-0003W5-Tx for bug-guix@gnu.org; Sat, 13 Nov 2021 16:39:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:36781) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mm0jm-0001is-Lm for bug-guix@gnu.org; Sat, 13 Nov 2021 16:39:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mm0jm-0003uK-KB for bug-guix@gnu.org; Sat, 13 Nov 2021 16:39:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#24937: [PATCH 2/2] daemon: Do not deduplicate files smaller than 4 KiB. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 13 Nov 2021 21:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24937 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 24937@debbugs.gnu.org Received: via spool by 24937-submit@debbugs.gnu.org id=B24937.163683949114948 (code B ref 24937); Sat, 13 Nov 2021 21:39:02 +0000 Received: (at 24937) by debbugs.gnu.org; 13 Nov 2021 21:38:11 +0000 Received: from localhost ([127.0.0.1]:48326 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mm0ip-0003sn-DX for submit@debbugs.gnu.org; Sat, 13 Nov 2021 16:38:11 -0500 Received: from eggs.gnu.org ([209.51.188.92]:45518) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mm0im-0003s9-RK for 24937@debbugs.gnu.org; Sat, 13 Nov 2021 16:38:02 -0500 Received: from [2001:470:142:3::e] (port=33782 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mm0ih-0001fC-GW; Sat, 13 Nov 2021 16:37:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=NY0T0saNoMKa7NZDLn1D9YXeU/MfWqRU6xnEnDT/Sv0=; b=nLJZHbDAJjLBzsTXPKke XcWzbEChEucPNAh5YjRWXTphvurzF0HzVho93bsdYYmnqyBVfPG4asIDLCSJEvjG4i9BURf6QjoB4 +5mnm5AGL9D35hnuj8/UlHBdF5+XbwRphtYpmkCqhStb1toPtOhvtqNS+uOMtjgFd7+xK0awAcgqp xqJ1OFhvTtTCByq0zwwTS6yZVQz0dd64mKY22zKyAvyqfpxj1nkihrsGgSwi/VhOdGT/3YjbNZ1rr UFxbHjOU7aZ6GkiFqM1gQ2+4S7fbjYcSrv6jQeD7eIS7/r792gyv9dIVBMgAoApRHXQASC2mIBKTc FDR6JJjdK+yfVw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:63350 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mm0ig-0000wh-7v; Sat, 13 Nov 2021 16:37:54 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sat, 13 Nov 2021 22:37:45 +0100 Message-Id: <20211113213745.2601-2-ludo@gnu.org> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211113213745.2601-1-ludo@gnu.org> References: <87v90wat9n.fsf@gnu.org> <20211113213745.2601-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1636839555; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=NY0T0saNoMKa7NZDLn1D9YXeU/MfWqRU6xnEnDT/Sv0=; b=IJM3HJz5W87Qixp1VdBDvMSahL+NcsUswEJjRCuV6Z88gNk4dWkRFtOFlXjWF3T7Z4Ga5R tGIpT46Dx+I++TMyjqMNDeztE1fKUBpDK6Y5CqHf3Dv75bHmkcDWBT9okcZ9ZVoxGTwkZ/ b63ItcqJg1jvQ2rl8R0cZaezNaZRnoViNNQcimhcUn29dA3x4U5V0ENEV06VvrLOx/N23O QVokGe6U3rYX2qFC7aSN2g2zK+YiP0SrINjjqBg/QXsYBsnRQFmy3gDK5nn3r2MADiPo9T EfLOdOAe22aIeAU7qjyw2lmxc01FYuTBNfo8ZmWQgVhxLda+PHhGeQJy3/UFTQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1636839555; a=rsa-sha256; cv=none; b=L06c741x9HqcoWpOlcdH7Q/T5nJvcv+ModdKyhK5kOjNvEHwMmw82VgUTiI3YH4xPpUAph YUemvUXzfSzKgeq2jqiOT5CZSDuGp+dMYwxzBtfME9/e5rzY/mPE335G4vIYnN25iHyGBp t5f2WN2tZ2L6obikZHX0cyR9RA0MYJsnVo4YWIKjTBTRoXJUzjkClkY1vLNzd//CoKnGDh VP5ObJCgHIsYU5tIswF0vZipYDJRMijnbi79Kt6gUz1Grjt2mwpyNjXfterXlc/jA829ly yZz/LizpXzRv7Mz1JKBEVLmxRZnPJVrW1oG9ZVWRIH1l175unKnWQ8H8NLZ8rA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=nLJZHbDA; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.04 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=nLJZHbDA; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 4629532A82 X-Spam-Score: -1.04 X-Migadu-Scanner: scn0.migadu.com X-TUID: L4OHw+jKjdWD Files smaller than 4 KiB typically represent ~60% of the entries in /gnu/store/.links but only contribute to ~2.5% of the space savings afforded by deduplication. Not considering these files for deduplication speeds up file insertion in the store and, more importantly, leaves 'removeUnusedLinks' with fewer entries to traverse, thereby speeding it up proportionally. Partly fixes . * config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK definition. * guix/store/deduplication.scm (%deduplication-minimum-size): New variable. (deduplicate)[loop]: Do not recurse when FILE's size is below %DEDUPLICATION-MINIMUM-SIZE. (dump-port): New procedure. (dump-file/deduplicate)[hash]: Turn into... [dump-and-compute-hash]: ... this thunk. Call 'deduplicate' only when SIZE is greater than %DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'. * nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where st.st_size < deduplicationMinSize. * nix/libstore/local-store.hh (deduplicationMinSize): New declaration. * nix/libstore/optimise-store.cc (deduplicationMinSize): New variable. (LocalStore::optimisePath_): Return when PATH is a symlink or smaller than 'deduplicationMinSize'. * tests/derivations.scm ("identical files are deduplicated"): Produce files bigger than %DEDUPLICATION-MINIMUM-SIZE. * tests/nar.scm ("restore-file-set with directories (signed, valid)"): Likewise. * tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"): New test. ("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than %DEDUPLICATION-MINIMUM-SIZE. * tests/store.scm ("substitute, deduplication"): Likewise. --- config-daemon.ac | 11 ------- guix/store/deduplication.scm | 57 ++++++++++++++++++++++++++++------ nix/libstore/gc.cc | 4 ++- nix/libstore/local-store.hh | 3 ++ nix/libstore/optimise-store.cc | 15 +++++---- tests/derivations.scm | 14 ++++++--- tests/nar.scm | 7 +++-- tests/store-deduplication.scm | 41 ++++++++++++++++++++---- tests/store.scm | 4 ++- 9 files changed, 114 insertions(+), 42 deletions(-) diff --git a/config-daemon.ac b/config-daemon.ac index 5ddc740600..86306effe1 100644 --- a/config-daemon.ac +++ b/config-daemon.ac @@ -94,17 +94,6 @@ if test "x$guix_build_daemon" = "xyes"; then AC_CHECK_FUNCS([lutimes lchown posix_fallocate sched_setaffinity \ statvfs nanosleep strsignal statx]) - dnl Check whether the store optimiser can optimise symlinks. - AC_MSG_CHECKING([whether it is possible to create a link to a symlink]) - ln -s bla tmp_link - if ln tmp_link tmp_link2 2> /dev/null; then - AC_MSG_RESULT(yes) - AC_DEFINE(CAN_LINK_SYMLINK, 1, [Whether link() works on symlinks.]) - else - AC_MSG_RESULT(no) - fi - rm -f tmp_link tmp_link2 - dnl Check for . AC_LANG_PUSH(C++) AC_CHECK_HEADERS([locale]) diff --git a/guix/store/deduplication.scm b/guix/store/deduplication.scm index cd9660174c..8a59adad39 100644 --- a/guix/store/deduplication.scm +++ b/guix/store/deduplication.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Caleb Ristvedt -;;; Copyright © 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2018-2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,12 +22,13 @@ (define-module (guix store deduplication) #:use-module (gcrypt hash) - #:use-module (guix build utils) + #:use-module ((guix build utils) #:hide (dump-port)) #:use-module (guix build syscalls) #:use-module (guix base32) #:use-module (srfi srfi-11) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) + #:use-module (rnrs bytevectors) #:use-module (rnrs io ports) #:use-module (ice-9 ftw) #:use-module (ice-9 match) @@ -37,6 +38,31 @@ (define-module (guix store deduplication) dump-file/deduplicate copy-file/deduplicate)) +;; TODO: Remove once 'dump-port' in (guix build utils) has an optional 'len' +;; parameter. +(define* (dump-port in out + #:optional len + #:key (buffer-size 16384)) + "Read LEN bytes from IN (or as much as possible if LEN is #f) and write it +to OUT, using chunks of BUFFER-SIZE bytes." + (define buffer + (make-bytevector buffer-size)) + + (let loop ((total 0) + (bytes (get-bytevector-n! in buffer 0 + (if len + (min len buffer-size) + buffer-size)))) + (or (eof-object? bytes) + (and len (= total len)) + (let ((total (+ total bytes))) + (put-bytevector out buffer 0 bytes) + (loop total + (get-bytevector-n! in buffer 0 + (if len + (min (- len total) buffer-size) + buffer-size))))))) + (define (nar-sha256 file) "Gives the sha256 hash of a file and the size of the file in nar form." (let-values (((port get-hash) (open-sha256-port))) @@ -127,6 +153,12 @@ (define temp-link (unless (= EMLINK (system-error-errno args)) (apply throw args))))))) +(define %deduplication-minimum-size + ;; Size below which files are not deduplicated. This avoids adding too many + ;; entries to '.links', which would slow down 'removeUnusedLinks' while + ;; saving little space. Keep in sync with optimize-store.cc. + 4096) + (define* (deduplicate path hash #:key (store (%store-directory))) "Check if a store item with sha256 hash HASH already exists. If so, replace PATH with a hardlink to the already-existing one. If not, register @@ -144,13 +176,16 @@ (define links-directory ((file . properties) (unless (member file '("." "..")) (let* ((file (string-append path "/" file)) + (st (lstat file)) (type (match (assoc-ref properties 'type) ((or 'unknown #f) - (stat:type (lstat file))) + (stat:type st)) (type type)))) - (loop file type - (and (not (eq? 'directory type)) - (nar-sha256 file))))))) + (unless (< (stat:size st) + %deduplication-minimum-size) + (loop file type + (and (not (eq? 'directory type)) + (nar-sha256 file)))))))) (scandir* path)) (let ((link-file (string-append links-directory "/" (bytevector->nix-base32-string hash)))) @@ -222,9 +257,9 @@ (define* (dump-file/deduplicate file input size type This procedure is suitable as a #:dump-file argument to 'restore-file'. When used that way, it deduplicates files on the fly as they are restored, thereby -removing the need to a deduplication pass that would re-read all the files +removing the need for a deduplication pass that would re-read all the files down the road." - (define hash + (define (dump-and-compute-hash) (call-with-output-file file (lambda (output) (let-values (((hash-port get-hash) @@ -236,7 +271,11 @@ (define hash (close-port hash-port) (get-hash))))) - (deduplicate file hash #:store store)) + (if (>= size %deduplication-minimum-size) + (deduplicate file (dump-and-compute-hash) #:store store) + (call-with-output-file file + (lambda (output) + (dump-port input output size))))) (define* (copy-file/deduplicate source target #:key (store (%store-directory))) diff --git a/nix/libstore/gc.cc b/nix/libstore/gc.cc index e1d0765154..16519116e4 100644 --- a/nix/libstore/gc.cc +++ b/nix/libstore/gc.cc @@ -606,7 +606,9 @@ void LocalStore::removeUnusedLinks(const GCState & state) throw SysError(format("statting `%1%'") % path); #endif - if (st.st_nlink != 1) { + /* Drop links for files smaller than 'deduplicationMinSize', even if + they have more than one hard link. */ + if (st.st_nlink != 1 && st.st_size >= deduplicationMinSize) { actualSize += st.st_size; unsharedSize += (st.st_nlink - 1) * st.st_size; continue; diff --git a/nix/libstore/local-store.hh b/nix/libstore/local-store.hh index 9ba37219da..20d3c3c893 100644 --- a/nix/libstore/local-store.hh +++ b/nix/libstore/local-store.hh @@ -292,4 +292,7 @@ void canonicaliseTimestampAndPermissions(const Path & path); MakeError(PathInUse, Error); +/* Size below which a file is not considered for deduplication. */ +extern const size_t deduplicationMinSize; + } diff --git a/nix/libstore/optimise-store.cc b/nix/libstore/optimise-store.cc index eb303ab4c3..baca1a4890 100644 --- a/nix/libstore/optimise-store.cc +++ b/nix/libstore/optimise-store.cc @@ -15,6 +15,9 @@ namespace nix { +/* Any file smaller than this is not considered for deduplication. + Keep in sync with (guix store deduplication). */ +const size_t deduplicationMinSize = 4096; static void makeWritable(const Path & path) { @@ -105,12 +108,12 @@ void LocalStore::optimisePath_(OptimiseStats & stats, const Path & path, InodeHa return; } - /* We can hard link regular files and maybe symlinks. */ - if (!S_ISREG(st.st_mode) -#if CAN_LINK_SYMLINK - && !S_ISLNK(st.st_mode) -#endif - ) return; + /* We can hard link regular files (and maybe symlinks), but do that only + for files larger than some threshold. This avoids adding too many + entries to '.links', which would slow down 'removeUnusedLinks' while + saving little space. */ + if (!S_ISREG(st.st_mode) || ((size_t) st.st_size) < deduplicationMinSize) + return; /* Sometimes SNAFUs can cause files in the store to be modified, in particular when running programs as root under diff --git a/tests/derivations.scm b/tests/derivations.scm index cd165d1be6..4621098df3 100644 --- a/tests/derivations.scm +++ b/tests/derivations.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2012-2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -170,11 +170,15 @@ (define prefix-len (string-length dir)) #f)))) (test-assert "identical files are deduplicated" - (let* ((build1 (add-text-to-store %store "one.sh" - "echo hello, world > \"$out\"\n" + ;; Note: DATA must be longer than %DEDUPLICATION-MINIMUM-SIZE. + (let* ((data (make-string 4500 #\a)) + (build1 (add-text-to-store %store "one.sh" + (string-append "echo -n " data + " > \"$out\"\n") '())) (build2 (add-text-to-store %store "two.sh" - "# Hey!\necho hello, world > \"$out\"\n" + (string-append "# Hey!\necho -n " + data " > \"$out\"\n") '())) (drv1 (derivation %store "foo" %bash `(,build1) @@ -187,7 +191,7 @@ (define prefix-len (string-length dir)) (file2 (derivation->output-path drv2))) (and (valid-path? %store file1) (valid-path? %store file2) (string=? (call-with-input-file file1 get-string-all) - "hello, world\n") + data) (= (stat:ino (lstat file1)) (stat:ino (lstat file2)))))))) diff --git a/tests/nar.scm b/tests/nar.scm index ba4881caaa..bd2bf6e6e0 100644 --- a/tests/nar.scm +++ b/tests/nar.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2012-2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -486,8 +486,9 @@ (define-values (port get-bytevector) ;; their mtime and permissions were not reset. Ensure that this bug is ;; gone. (with-store store - (let* ((text1 (random-text)) - (text2 (random-text)) + ;; Note: TEXT1 and TEXT2 must be longer than %DEDUPLICATION-MINIMUM-SIZE. + (let* ((text1 (string-concatenate (make-list 100 (random-text)))) + (text2 (string-concatenate (make-list 100 (random-text)))) (tree `("tree" directory ("a" regular (data ,text1)) ("b" directory diff --git a/tests/store-deduplication.scm b/tests/store-deduplication.scm index b1c2d93bbd..b2b7c36622 100644 --- a/tests/store-deduplication.scm +++ b/tests/store-deduplication.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2018, 2020 Ludovic Courtès +;;; Copyright © 2018, 2020-2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -30,13 +30,40 @@ (define-module (test-store-deduplication) (test-begin "store-deduplication") +(test-equal "deduplicate, below %deduplication-minimum-size" + (list #t (make-list 5 1)) + + (call-with-temporary-directory + (lambda (store) + ;; Note: DATA must be longer than %DEDUPLICATION-MINIMUM-SIZE. + (let ((data "Hello, world!") + (identical (map (lambda (n) + (string-append store "/" (number->string n) + "/a/b/c")) + (iota 5)))) + (for-each (lambda (file) + (mkdir-p (dirname file)) + (call-with-output-file file + (lambda (port) + (put-bytevector port (string->utf8 data))))) + identical) + + (deduplicate store (nar-sha256 store) #:store store) + + ;; (system (string-append "ls -lRia " store)) + (list (= (length (delete-duplicates + (map (compose stat:ino stat) identical))) + (length identical)) + (map (compose stat:nlink stat) identical)))))) + (test-equal "deduplicate" (cons* #t #f ;inode comparisons 2 (make-list 5 6)) ;'nlink' values (call-with-temporary-directory (lambda (store) - (let ((data (string->utf8 "Hello, world!")) + ;; Note: DATA must be longer than %DEDUPLICATION-MINIMUM-SIZE. + (let ((data (string-concatenate (make-list 500 "Hello, world!"))) (identical (map (lambda (n) (string-append store "/" (number->string n) "/a/b/c")) @@ -46,7 +73,7 @@ (define-module (test-store-deduplication) (mkdir-p (dirname file)) (call-with-output-file file (lambda (port) - (put-bytevector port data)))) + (put-bytevector port (string->utf8 data))))) identical) ;; Make the parent of IDENTICAL read-only. This should not prevent ;; deduplication from inserting its hard link. @@ -54,7 +81,7 @@ (define-module (test-store-deduplication) (call-with-output-file unique (lambda (port) - (put-bytevector port (string->utf8 "This is unique.")))) + (put-bytevector port (string->utf8 (string-reverse data))))) (deduplicate store (nar-sha256 store) #:store store) @@ -77,8 +104,10 @@ (define-module (test-store-deduplication) (lambda (store) (let ((true-link link) (links 0) - (data1 (string->utf8 "Hello, world!")) - (data2 (string->utf8 "Hi, world!")) + (data1 (string->utf8 + (string-concatenate (make-list 500 "Hello, world!")))) + (data2 (string->utf8 + (string-concatenate (make-list 500 "Hi, world!")))) (identical (map (lambda (n) (string-append store "/" (number->string n) "/a/b/c")) diff --git a/tests/store.scm b/tests/store.scm index 2150a0048c..5089909362 100644 --- a/tests/store.scm +++ b/tests/store.scm @@ -759,7 +759,9 @@ (define lst (test-assert "substitute, deduplication" (with-store s - (let* ((c (random-text)) ; contents of the output + ;; Note: C must be longer than %DEDUPLICATION-MINIMUM-SIZE. + (let* ((c (string-concatenate + (make-list 100 (random-text)))) ; contents of the output (g (package-derivation s %bootstrap-guile)) (d1 (build-expression->derivation s "substitute-me" `(begin ,c (exit 1)) -- 2.33.0