From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QGv5BSYiFGDQIAAA0tVLHw (envelope-from ) for ; Fri, 29 Jan 2021 14:56:38 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id wNG3ASYiFGCkTgAAbx9fmQ (envelope-from ) for ; Fri, 29 Jan 2021 14:56:38 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B9E179403C9 for ; Fri, 29 Jan 2021 14:56:37 +0000 (UTC) Received: from localhost ([::1]:47984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5VCO-0001oU-La for larch@yhetil.org; Fri, 29 Jan 2021 09:56:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:49740) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5VBl-0001nW-05 for guix-devel@gnu.org; Fri, 29 Jan 2021 09:56:00 -0500 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]:41691) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l5VBh-00067G-Ta for guix-devel@gnu.org; Fri, 29 Jan 2021 09:55:55 -0500 Received: by mail-wr1-x42c.google.com with SMTP id p15so9102642wrq.8 for ; Fri, 29 Jan 2021 06:55:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=tt2233/iLRREbxnfaVuh6AWhdNd0G/jILrCp38L2A1U=; b=rrGYPR3A3B+9nvbCJr0PjPcM40j/xJyER+c41Cu6PP02+pPTVHOM2cfBN8Iu/8fg2N Sj7BHhP96CCx/WsdByyhNzUV3w3qavfhWcUYjHdysidR9UJ0cUuOGdRESObjaWvgZ11i OnZa5Jrj9AVZGC9B4b473WltIYVlMO4TlXkZeQtqKyFzbpq/vDHlPXC/MLTK+TAQsoEy WNkQJZUYfVutdEozU83mWUyHPAB50es+R/xSskxMxDEabBu5hFGeghMA/hLpTQd0n68g CnZo/Gi5jGvHbFFwPY6ox9QuAGYMCs3qdwjFKHXG/ioLzT65OW758Fi17katEZVI0Cn4 M3+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=tt2233/iLRREbxnfaVuh6AWhdNd0G/jILrCp38L2A1U=; b=g+wG6Pv4sOt0j9nFEM9wnkoYpUHZEiZaIqBX0vYlDP56L1PaFTBZj8QcEsSrmexqFO 8afr3aedFIA2b6IuIuhDsbPrIStLZzvRI9qoqGTsZGci7LXg0IN/UBM4gL47AbEG9QEn 3xVSwO+fnd7QOt4jVDOXhcuDIoeZh1lm7r/PZ4HScz2OB95E18IJCJ+jaMR7Is8dFXfd CdxojCdon6uhiwR00nI9I+cc3nuXaorlhTJs96rAGU28Nvs5Tv/X0kONs3Q50oVnPcEZ J6nvelA7yy49aOTArNFU1/YsPVFQ/zdAHYQqSVslA4/0QQmYaP87Bqs0WDK6uh0WXdTR pfoQ== X-Gm-Message-State: AOAM533VMo15MABX/oFt0xA0QL/obqoGakwncR+Vmje7J2PMXoB4kXHc gQCISg6cQ6NddE0J6CjZdt4mqaG39I4= X-Google-Smtp-Source: ABdhPJyKLTeteJPYiUSAJsr+ikcki6n08SeAbFJkb2FewKdF2Fsk4C58bD2r2PDL10lnaUN8rAUeRA== X-Received: by 2002:a5d:452f:: with SMTP id j15mr5095061wra.298.1611932152350; Fri, 29 Jan 2021 06:55:52 -0800 (PST) Received: from guixSD (host-87-4-165-57.retail.telecomitalia.it. [87.4.165.57]) by smtp.gmail.com with ESMTPSA id z8sm12950317wrh.83.2021.01.29.06.55.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Jan 2021 06:55:51 -0800 (PST) From: =?utf-8?Q?Nicol=C3=B2?= Balzarotti To: Pierre Neidhardt , Guillaume Le Vaillant Subject: Re: When substitute download + decompression is CPU-bound In-Reply-To: <87lfccrley.fsf@ambrevar.xyz> References: <87im94qbby.fsf@gnu.org> <94405d66-b13c-e6e6-e8d5-df23b93e5d97@web.de> <87im92voqw.fsf@dismail.de> <87ft3d2fge.fsf@yamatai> <87bldr191v.fsf@gnu.org> <87h7ni62nt.fsf@ambrevar.xyz> <87bld9j651.fsf@gnu.org> <87eei4rs8x.fsf@ambrevar.xyz> <874kj0x9yv.fsf@yamatai> <87mtwsarfg.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <87lfccrley.fsf@ambrevar.xyz> Date: Fri, 29 Jan 2021 15:55:49 +0100 Message-ID: <87bld7bxmy.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2a00:1450:4864:20::42c; envelope-from=anothersms@gmail.com; helo=mail-wr1-x42c.google.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_BTC_ID=0.498, PDS_LITECOIN_ID=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -3.05 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=rrGYPR3A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: B9E179403C9 X-Spam-Score: -3.05 X-Migadu-Scanner: scn0.migadu.com X-TUID: 7sgHX+GTIy/n --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Pierre Neidhardt writes: > Nicol=C3=B2 Balzarotti writes: > > What wouldn't be the case? If you mean that "gzip is never the best > choice", wouldn't Zstd outperform gzip on the Raspberry Pi 1 too? My bad, you are right. Also, memory usage shoudn't be a problem. gzip uses way less (testd on ungoogled chromium, I get ~16kb peak heap size for gzip, 8Mb for zstd and 32Mb for lzip), but I'd expect guix to be running on systems with more than 8Mb of memory. Just for reference, here's the memory profiling script --=-=-= Content-Type: text/x-org Content-Disposition: inline; filename=prof.org Content-Description: profiling #+PROPERTY: header-args:bash :session valgrind * NAR Decompression memory benchmark #+begin_src bash :results none guix environment --ad-hoc valgrind lzip gzip zstd wget #+end_src #+begin_src bash :cache yes valgrind --version | sed 's/-/ /' lzip --version | head -1 gunzip --version | head -1 | sed 's/\s(/(/' zstd --version | sed -e 's/command.*v//' -e 's/,.*//' -e 's/**//' #+end_src #+RESULTS[e07cecfd5cc770b7a898408b80678f2e8ea7772e]: | valgrind | 3.16.1 | | lzip | 1.21 | | gunzip(gzip) | 1.1 | | zstd | 1.4.4 | Just noticed that there should be a new zstd release ([[https://github.com/facebook/zstd/releases/][zstd 1.4.8]]), and a new lzip release ([[https://download.savannah.gnu.org/releases/lzip/][lzip 1.22]]). ** Prepare required data #+begin_src bash :cache yes wget https://ci.guix.gnu.org/nar/gzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92 -O uc.nar.gz wget https://ci.guix.gnu.org/nar/lzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92 -O uc.nar.lz #+end_src #+RESULTS[ea17e5a54da1ca54a9c82f264912675d9ca981a0]: Create zstd compressed file #+begin_src bash :results none gunzip -c < uc.nar.gz > uc.nar zstd -19 uc.nar -o uc.nar.zstd #+end_src Check file sizes #+begin_src bash ls -lh --sort=size | head -5 #+end_src #+RESULTS: | total | 585M | | | | | | | | | -rw-r--r-- | 1 | nixo | users | 335M | Jan | 29 | 15:14 | uc.nar | | -rw-r--r-- | 1 | nixo | users | 103M | Jan | 29 | 15:13 | uc.nar.gz | | -rw-r--r-- | 1 | nixo | users | 78M | Jan | 29 | 15:14 | uc.nar.zstd | | -rw-r--r-- | 1 | nixo | users | 71M | Jan | 29 | 15:13 | uc.nar.lz | ** Decompress #+name: massif #+begin_src bash :session valgrind :var command="ls" input="." output="/dev/null" name="ls" time valgrind --tool=massif --log-file=/dev/null --time-unit=B --trace-children=yes --massif-out-file=$name.massif $command < $input >$output #+end_src #+call: massif(command="gunzip -c", input="uc.nar.gz", output="/dev/null", name="gzip") #+RESULTS: | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | | real | 0m8.291s | | | | | | | | | user | 0m7.910s | | | | | | | | | sys | 0m0.201s | | | | | | | | #+call: massif(command="lzip -d", input="uc.nar.lz", output="/dev/null", name="lzip") #+RESULTS: | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | | real | 0m22.378s | | | | | | | | | user | 0m20.959s | | | | | | | | | sys | 0m0.345s | | | | | | | | #+call: massif(command="zstd -d", input="uc.nar.zstd", output="/dev/null", name="zstd") #+RESULTS: | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ | | real | 0m4.607s | | | | | | | | | user | 0m4.157s | | | | | | | | | sys | 0m0.135s | | | | | | | | ** Check massif output #+begin_src bash :results raw drawer for ext in gzip lzip zstd; do ms_print $ext.massif > $ext.graph done #+end_src #+RESULTS: :results: :end: -------------------------------------------------------------------------------- Command: /gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin/gzip -d -c Massif arguments: --time-unit=B --massif-out-file=gzip.massif ms_print arguments: gzip.massif -------------------------------------------------------------------------------- KB 15.59^ # | # | # | # | # | # : | # : | # : | : # :: : : | : # :: : : : | : :#: ::::: :: : :: : : : : : @@ : | : :#: :: : :: : : : : : : : @ : | : :#: :: : :: : : : : : : : @ : | : :#: :: : :: : : : : : : : : @ : | : :#: :: : :: : : : : : : : : @ : | : :#: :: : ::: ::: :::: : : : :: : @ : : | : :#: :: : ::: ::: :: :: : : : :::: :::: : @ : : | : :#::::: : ::: ::: : :: : ::: : ::: : ::: :::@@ : @ ::: : | : :#:: :: : :::::::: : :: : :: :: ::: : ::: :: @ :::@ :: : | ::::#:: :: : :::: ::: : :: : :: :: ::: : ::: :: @ :::@ :: : : 0 +----------------------------------------------------------------------->MB 0 114.8 Number of snapshots: 53 Detailed snapshots: [5 (peak), 21, 39, 42, 46, 50] -------------------------------------------------------------------------------- Command: lzip -d Massif arguments: --time-unit=B --massif-out-file=lzip.massif ms_print arguments: lzip.massif -------------------------------------------------------------------------------- MB 32.09^ ################################### | # | # | # | # | # | # | # | # | # | # | # | # | # | # | # | # | # | # | # 0 +----------------------------------------------------------------------->MB 0 64.18 Number of snapshots: 12 Detailed snapshots: [6 (peak)] -------------------------------------------------------------------------------- Command: zstd -d Massif arguments: --time-unit=B --massif-out-file=zstd.massif ms_print arguments: zstd.massif -------------------------------------------------------------------------------- MB 8.665^ # | :#::::::::::::::::::::::::::::::::: | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# | :# 0 +----------------------------------------------------------------------->MB 0 17.33 Number of snapshots: 18 Detailed snapshots: [9 (peak)] --=-=-=--