From: "Ludovic Courtès" <ludo@gnu.org>
To: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Cc: 24937@debbugs.gnu.org
Subject: bug#24937: "deleting unused links" GC phase is too slow
Date: Sat, 13 Nov 2021 17:56:52 +0100 [thread overview]
Message-ID: <87v90wat9n.fsf@gnu.org> (raw)
In-Reply-To: <87v90ys911.fsf@gmail.com> (Maxim Cournoyer's message of "Thu, 11 Nov 2021 15:59:54 -0500")
[-- Attachment #1: Type: text/plain, Size: 630 bytes --]
Hi,
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> I haven't done any analysis, just grabbed the result, but here it what
> it looks for me:
There’s a bit more than 35% of deduplicated files that are < 1KiB, and
not much to be gained by deduplicating them.
On IRC several people shared the results on their machine; several had
similar results, and one person had a lot more of those small files (50%
of deduplicated files were < 1KiB).
The chart (with a kinda bogus layout) below is perhaps more interesting:
it shows the contribution of files below a certain size to the overall
space savings.
[-- Attachment #2: the chart --]
[-- Type: image/png, Size: 22673 bytes --]
[-- Attachment #3: Type: text/plain, Size: 2360 bytes --]
In a nutshell:
• Files < 1KiB contribute to 0.3% of the space savings;
• Files < 4KiB contribute to 2.5% of the space savings;
• Files < 256KiB contribute to 42% of the space savings.
You can create this plot with:
--8<---------------cut here---------------start------------->8---
(make-scatter-plot #:title "Contribution to space savings"
#:write-to-png "/tmp/space-saving-contribution.png"
#:chart-width 1000
#:y-axis-label "contribution (%)"
#:x-axis-label "size (B)"
#:log-x-base 2
#:min-x 513
#:data
(let ((total (saved-space l)))
`(("contribution"
,@(map (lambda (size)
(cons size
(/ (saved-space (filter (lambda (file)
(< (deduplicated-file-size
file)
size))
l))
total .01)))
(map (cut expt 2 <>)
(iota 12 10 1)))))))
--8<---------------cut here---------------end--------------->8---
You can also compute individual points like this:
--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (/ (saved-space (filter (lambda (file)
(< (deduplicated-file-size file) 1024))
l))
(saved-space l) 1.)
$60 = 0.0034284626558736746
scheme@(guile-user)> (/ (saved-space (filter (lambda (file)
(< (deduplicated-file-size file) 4096))
l))
(saved-space l) 1.)
$62 = 0.025190871178467848
scheme@(guile-user)> (/ (saved-space (filter (lambda (file)
(< (deduplicated-file-size file) (expt 2 18)))
l))
(saved-space l) 1.)
$65 = 0.42411104869782185
--8<---------------cut here---------------end--------------->8---
Choosing a deduplication threshold of 2KiB or 4KiB would have a
negligible impact on disk usage on my machine.
Thanks,
Ludo’.
next prev parent reply other threads:[~2021-11-13 16:58 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-13 17:41 bug#24937: "deleting unused links" GC phase is too slow Ludovic Courtès
2016-12-09 22:43 ` Ludovic Courtès
2016-12-11 13:46 ` Ludovic Courtès
2016-12-11 14:23 ` Mark H Weaver
2016-12-11 18:02 ` Ludovic Courtès
2016-12-11 19:27 ` Mark H Weaver
2016-12-13 0:00 ` Ludovic Courtès
2016-12-13 12:48 ` Mark H Weaver
2016-12-13 17:02 ` Ludovic Courtès
2016-12-13 17:18 ` Ricardo Wurmus
2020-04-16 13:26 ` Ricardo Wurmus
2020-04-16 14:27 ` Ricardo Wurmus
2020-04-17 8:16 ` Ludovic Courtès
2020-04-17 8:28 ` Ricardo Wurmus
2016-12-13 4:09 ` Mark H Weaver
2016-12-15 1:19 ` Mark H Weaver
2021-11-09 14:44 ` Ludovic Courtès
2021-11-09 15:00 ` Ludovic Courtès
2021-11-11 20:59 ` Maxim Cournoyer
2021-11-13 16:56 ` Ludovic Courtès [this message]
2021-11-13 21:37 ` bug#24937: [PATCH 1/2] tests: Factorize 'file=?' Ludovic Courtès
2021-11-13 21:37 ` bug#24937: [PATCH 2/2] daemon: Do not deduplicate files smaller than 4 KiB Ludovic Courtès
2021-11-16 13:54 ` bug#24937: "deleting unused links" GC phase is too slow Ludovic Courtès
2021-11-13 21:45 ` Ludovic Courtès
2021-11-22 2:30 ` John Kehayias via Bug reports for GNU Guix
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87v90wat9n.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=24937@debbugs.gnu.org \
--cc=maxim.cournoyer@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).