From: Guillaume Le Vaillant <glv@posteo.net>
To: Joshua Branson <jbranso@dismail.de>
Cc: guix-devel@gnu.org
Subject: Re: When substitute download + decompression is CPU-bound
Date: Thu, 07 Jan 2021 11:45:21 +0100 [thread overview]
Message-ID: <87ft3d2fge.fsf@yamatai> (raw)
In-Reply-To: <87im92voqw.fsf@dismail.de>
[-- Attachment #1.1: Type: text/plain, Size: 353 bytes --]
I compared gzip, lzip and zstd when compressing a 580 MB pack (therefore
containing "subsitutes" for several packages) with different compression
levels. Maybe the results can be of some use to someone.
Note that the plots only show the results using only 1 thread and
standard compression levels, and that the speed axis is using
logarithmic scale.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: compression-benchmark.org --]
[-- Type: text/x-org, Size: 9159 bytes --]
Machine used for the tests:
- CPU: Intel i7-3630QM
- RAM: 16 MiB
Programs:
- gzip 1.10
- pigz 2.4
- lzip 1.21
- plzip 1.8
- zstd 1.4.4
- pzstd 1.4.4
Uncompressed file:
- name: monero-0.17.1.5-pack.tar
- size: 582707200 bytes
#+PLOT: script:"compression-benchmark.plot"
| Comp. command | Comp. time | Comp. size | Comp. speed | Comp. ratio | Decomp. time | Decomp. speed |
|----------------+------------+------------+-------------+-------------+--------------+---------------|
| gzip -1 | 7.999 | 166904534 | 72847506 | 3.491 | 3.292 | 50700041 |
| gzip -2 | 8.469 | 161859128 | 68804723 | 3.600 | 3.214 | 50360650 |
| gzip -3 | 10.239 | 157839772 | 56910558 | 3.692 | 3.144 | 50203490 |
| gzip -4 | 11.035 | 151039457 | 52805365 | 3.858 | 3.104 | 48659619 |
| gzip -5 | 13.767 | 146693142 | 42326375 | 3.972 | 3.143 | 46672969 |
| gzip -6 | 19.707 | 144364588 | 29568539 | 4.036 | 3.001 | 48105494 |
| gzip -7 | 24.014 | 143727357 | 24265312 | 4.054 | 2.993 | 48021168 |
| gzip -8 | 43.219 | 143062985 | 13482663 | 4.073 | 2.969 | 48185579 |
| gzip -9 | 70.930 | 142803637 | 8215243 | 4.080 | 2.964 | 48179365 |
| pigz -1 -p 4 | 2.247 | 165745308 | 259326747 | 3.516 | 1.919 | 86370666 |
| pigz -2 -p 4 | 2.394 | 160661935 | 243403175 | 3.627 | 1.862 | 86284605 |
| pigz -3 -p 4 | 2.776 | 156696382 | 209908934 | 3.719 | 1.817 | 86239065 |
| pigz -4 -p 4 | 3.045 | 150539955 | 191365255 | 3.871 | 1.787 | 84241721 |
| pigz -5 -p 4 | 3.855 | 146289903 | 151156213 | 3.983 | 1.732 | 84462992 |
| pigz -6 -p 4 | 5.378 | 143967093 | 108350167 | 4.048 | 1.721 | 83653163 |
| pigz -7 -p 4 | 6.579 | 143350506 | 88570786 | 4.065 | 1.702 | 84224739 |
| pigz -8 -p 4 | 11.76 | 142738270 | 49549932 | 4.082 | 1.720 | 82987366 |
| pigz -9 -p 4 | 19.878 | 142479078 | 29314176 | 4.090 | 1.691 | 84257290 |
| lzip -0 | 16.686 | 130302649 | 34921923 | 4.472 | 9.981 | 13055070 |
| lzip -1 | 42.011 | 118070414 | 13870348 | 4.935 | 8.669 | 13619842 |
| lzip -2 | 51.395 | 112769303 | 11337819 | 5.167 | 8.368 | 13476255 |
| lzip -3 | 69.344 | 106182860 | 8403138 | 5.488 | 8.162 | 13009417 |
| lzip -4 | 89.781 | 100072461 | 6490318 | 5.823 | 7.837 | 12769231 |
| lzip -5 | 119.626 | 95033235 | 4871075 | 6.132 | 7.586 | 12527450 |
| lzip -6 | 155.740 | 83063613 | 3741538 | 7.015 | 6.856 | 12115463 |
| lzip -7 | 197.485 | 78596381 | 2950640 | 7.414 | 6.586 | 11933857 |
| lzip -8 | 238.076 | 72885403 | 2447568 | 7.995 | 6.227 | 11704738 |
| lzip -9 | 306.368 | 72279340 | 1901985 | 8.062 | 6.203 | 11652320 |
| plzip -0 -n 4 | 4.821 | 131211238 | 120868533 | 4.441 | 2.829 | 46380784 |
| plzip -1 -n 4 | 13.453 | 120565830 | 43314294 | 4.833 | 2.604 | 46300242 |
| plzip -2 -n 4 | 15.695 | 114874773 | 37126932 | 5.073 | 2.398 | 47904409 |
| plzip -3 -n 4 | 20.563 | 108896468 | 28337655 | 5.351 | 2.486 | 43803889 |
| plzip -4 -n 4 | 26.871 | 102285879 | 21685356 | 5.697 | 2.375 | 43067739 |
| plzip -5 -n 4 | 35.220 | 97402840 | 16544781 | 5.982 | 2.448 | 39788742 |
| plzip -6 -n 4 | 45.812 | 89260273 | 12719532 | 6.528 | 2.145 | 41613181 |
| plzip -7 -n 4 | 62.723 | 82944080 | 9290168 | 7.025 | 2.080 | 39876962 |
| plzip -8 -n 4 | 71.928 | 78477272 | 8101257 | 7.425 | 2.120 | 37017581 |
| plzip -9 -n 4 | 103.744 | 75648923 | 5616780 | 7.703 | 2.578 | 29344035 |
| zstd -1 | 2.057 | 145784609 | 283280117 | 3.997 | 0.639 | 228144928 |
| zstd -2 | 2.316 | 136049621 | 251600691 | 4.283 | 0.657 | 207077049 |
| zstd -3 | 2.733 | 127702753 | 213211562 | 4.563 | 0.650 | 196465774 |
| zstd -4 | 3.269 | 126224007 | 178252432 | 4.616 | 0.658 | 191829798 |
| zstd -5 | 5.136 | 122024478 | 113455452 | 4.775 | 0.680 | 179447762 |
| zstd -6 | 6.394 | 120035201 | 91133438 | 4.854 | 0.652 | 184103069 |
| zstd -7 | 8.510 | 116048780 | 68473231 | 5.021 | 0.612 | 189622190 |
| zstd -8 | 9.875 | 114821611 | 59008324 | 5.075 | 0.593 | 193628349 |
| zstd -9 | 12.478 | 113868149 | 46698766 | 5.117 | 0.588 | 193653315 |
| zstd -10 | 14.982 | 111113753 | 38893819 | 5.244 | 0.578 | 192238327 |
| zstd -11 | 16.391 | 110674252 | 35550436 | 5.265 | 0.583 | 189835767 |
| zstd -12 | 21.008 | 110031164 | 27737395 | 5.296 | 0.570 | 193037130 |
| zstd -13 | 51.259 | 109262475 | 11367900 | 5.333 | 0.561 | 194763770 |
| zstd -14 | 58.897 | 108632734 | 9893665 | 5.364 | 0.562 | 193296680 |
| zstd -15 | 82.514 | 107956132 | 7061919 | 5.398 | 0.557 | 193817113 |
| zstd -16 | 78.935 | 105533404 | 7382114 | 5.522 | 0.576 | 183217715 |
| zstd -17 | 89.832 | 94165409 | 6486633 | 6.188 | 0.565 | 166664441 |
| zstd -18 | 115.663 | 91124039 | 5037974 | 6.395 | 0.614 | 148410487 |
| zstd -19 | 157.008 | 90229137 | 3711322 | 6.458 | 0.614 | 146952992 |
| zstd -20 | 162.499 | 80742922 | 3585913 | 7.217 | 0.605 | 133459375 |
| zstd -21 | 207.122 | 79619348 | 2813353 | 7.319 | 0.611 | 130309899 |
| zstd -22 | 277.177 | 78652901 | 2102293 | 7.409 | 0.634 | 124058203 |
| pzstd -1 -p 4 | 0.621 | 146665510 | 938336876 | 3.973 | 0.196 | 748293418 |
| pzstd -2 -p 4 | 0.720 | 137416958 | 809315556 | 4.240 | 0.227 | 605361048 |
| pzstd -3 -p 4 | 1.180 | 128748806 | 493819661 | 4.526 | 0.231 | 557354139 |
| pzstd -4 -p 4 | 1.786 | 127373154 | 326263830 | 4.575 | 0.240 | 530721475 |
| pzstd -5 -p 4 | 2.635 | 123216422 | 221141252 | 4.729 | 0.240 | 513401758 |
| pzstd -6 -p 4 | 3.774 | 121257316 | 154400424 | 4.806 | 0.251 | 483096876 |
| pzstd -7 -p 4 | 3.988 | 117361187 | 146115145 | 4.965 | 0.263 | 446240255 |
| pzstd -8 -p 4 | 4.540 | 116172098 | 128349604 | 5.016 | 0.240 | 484050408 |
| pzstd -9 -p 4 | 5.083 | 115237287 | 114638442 | 5.057 | 0.268 | 429989877 |
| pzstd -10 -p 4 | 5.630 | 112359994 | 103500391 | 5.186 | 0.226 | 497168115 |
| pzstd -11 -p 4 | 5.991 | 111969711 | 97263762 | 5.204 | 0.246 | 455161427 |
| pzstd -12 -p 4 | 8.001 | 111326376 | 72829296 | 5.234 | 0.227 | 490424564 |
| pzstd -13 -p 4 | 16.035 | 110525395 | 36339707 | 5.272 | 0.259 | 426738977 |
| pzstd -14 -p 4 | 18.145 | 109957500 | 32113927 | 5.299 | 0.253 | 434614625 |
| pzstd -15 -p 4 | 24.791 | 109358520 | 23504788 | 5.328 | 0.224 | 488207679 |
| pzstd -16 -p 4 | 23.940 | 106888588 | 24340317 | 5.452 | 0.234 | 456788838 |
| pzstd -17 -p 4 | 29.099 | 97393935 | 20024991 | 5.983 | 0.266 | 366142613 |
| pzstd -18 -p 4 | 37.124 | 94273955 | 15696240 | 6.181 | 0.284 | 331950546 |
| pzstd -19 -p 4 | 48.798 | 93531545 | 11941211 | 6.230 | 0.262 | 356990630 |
| pzstd -20 -p 4 | 54.860 | 82067608 | 10621713 | 7.100 | 0.302 | 271747046 |
| pzstd -21 -p 4 | 64.179 | 79735488 | 9079406 | 7.308 | 0.389 | 204975548 |
| pzstd -22 -p 4 | 256.242 | 78688788 | 2274050 | 7.405 | 0.585 | 134510749 |
#+TBLFM: $4='(format "%d" (round (/ 582707200.0 $2)));N :: $5='(format "%.3f" (/ 582707200.0 $3));N :: $7='(format "%d" (round (/ $3 $6)));N
[-- Attachment #1.3: compression-benchmark.plot --]
[-- Type: text/plain, Size: 1631 bytes --]
set terminal png size 1920, 1080
set style data linespoints
set logscale y
set xlabel "Compression ratio"
set ylabel "Compression speed (MB/s)"
set output "compression.png"
plot '$datafile' every ::0::8 using 5:($4 / 1000000) linecolor "dark-violet" title "gzip", \
'$datafile' every ::0::8 using 5:($4 / 1000000):(substr(stringcolumn(1), 7, 8)) with labels textcolor "dark-violet" offset -1, -1 notitle, \
'$datafile' every ::18::27 using 5:($4 / 1000000) linecolor "navy" title "lzip", \
'$datafile' every ::18::27 using 5:($4 / 1000000):(substr(stringcolumn(1), 7, 8)) with labels textcolor "navy" offset 0, -1 notitle, \
'$datafile' every ::38::56 using 5:($4 / 1000000) linecolor "olive" title "zstd", \
'$datafile' every ::38::56 using 5:($4 / 1000000):(substr(stringcolumn(1), 7, 9)) with labels textcolor "olive" offset 1, 1 notitle
set ylabel "Decompression speed (MB/s)"
set output "decompression.png"
plot '$datafile' every ::0::8 using 5:($7 / 1000000) linecolor "dark-violet" title "gzip", \
'$datafile' every ::0::8 using 5:($7 / 1000000):(substr(stringcolumn(1), 7, 8)) with labels textcolor "dark-violet" offset 0, -1 notitle, \
'$datafile' every ::18::27 using 5:($7 / 1000000) linecolor "navy" title "lzip", \
'$datafile' every ::18::27 using 5:($7 / 1000000):(substr(stringcolumn(1), 7, 8)) with labels textcolor "navy" offset 0, -1 notitle, \
'$datafile' every ::38::56 using 5:($7 / 1000000) linecolor "olive" title "zstd", \
'$datafile' every ::38::56 using 5:($7 / 1000000):(substr(stringcolumn(1), 7, 8)) with labels textcolor "olive" offset 0, -1 notitle
[-- Attachment #1.4: compression.png --]
[-- Type: image/png, Size: 16056 bytes --]
[-- Attachment #1.5: decompression.png --]
[-- Type: image/png, Size: 12804 bytes --]
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]
next prev parent reply other threads:[~2021-01-07 10:46 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-14 22:20 When substitute download + decompression is CPU-bound Ludovic Courtès
2020-12-14 22:29 ` Julien Lepiller
2020-12-14 22:59 ` Nicolò Balzarotti
2020-12-15 7:52 ` Pierre Neidhardt
2020-12-15 9:45 ` Nicolò Balzarotti
2020-12-15 9:54 ` Pierre Neidhardt
2020-12-15 10:03 ` Nicolò Balzarotti
2020-12-15 10:13 ` Pierre Neidhardt
2020-12-15 10:14 ` Pierre Neidhardt
2020-12-15 11:42 ` Ludovic Courtès
2020-12-15 12:31 ` Pierre Neidhardt
2020-12-18 14:59 ` Ludovic Courtès
2020-12-18 15:33 ` Pierre Neidhardt
2020-12-15 11:36 ` Ludovic Courtès
2020-12-15 11:45 ` Nicolò Balzarotti
2020-12-15 10:40 ` Jonathan Brielmaier
2020-12-15 19:43 ` Joshua Branson
2021-01-07 10:45 ` Guillaume Le Vaillant [this message]
2021-01-07 11:00 ` Pierre Neidhardt
2021-01-07 11:33 ` Guillaume Le Vaillant
2021-01-14 21:51 ` Ludovic Courtès
2021-01-14 22:08 ` Nicolò Balzarotti
2021-01-28 17:53 ` Are gzip-compressed substitutes still used? Ludovic Courtès
2021-03-17 17:12 ` Ludovic Courtès
2021-03-17 17:33 ` Léo Le Bouter
2021-03-17 18:08 ` Vagrant Cascadian
2021-03-18 0:03 ` zimoun
2021-03-18 16:00 ` Vagrant Cascadian
2021-03-18 18:53 ` Leo Famulari
2021-03-20 11:23 ` Ludovic Courtès
2021-03-17 18:06 ` zimoun
2021-03-17 18:20 ` Jonathan Brielmaier
2021-03-18 17:25 ` Pierre Neidhardt
2021-01-15 8:10 ` When substitute download + decompression is CPU-bound Pierre Neidhardt
2021-01-28 17:58 ` Ludovic Courtès
2021-01-29 9:45 ` Pierre Neidhardt
2021-01-29 11:23 ` Guillaume Le Vaillant
2021-01-29 11:55 ` Nicolò Balzarotti
2021-01-29 12:13 ` Pierre Neidhardt
2021-01-29 13:06 ` Guillaume Le Vaillant
2021-01-29 14:55 ` Nicolò Balzarotti
2021-02-01 22:18 ` Ludovic Courtès
2021-01-29 13:33 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ft3d2fge.fsf@yamatai \
--to=glv@posteo.net \
--cc=guix-devel@gnu.org \
--cc=jbranso@dismail.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).