From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andreas Enge Subject: Parallel (de-)compression Date: Wed, 2 Dec 2015 19:45:50 +0100 Message-ID: <20151202184550.GA5741@debian> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35183) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a4CPc-00081i-5c for guix-devel@gnu.org; Wed, 02 Dec 2015 13:45:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a4CPZ-0006cA-0a for guix-devel@gnu.org; Wed, 02 Dec 2015 13:45:56 -0500 Received: from mout.kundenserver.de ([212.227.126.134]:49712) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a4CPY-0006c5-O7 for guix-devel@gnu.org; Wed, 02 Dec 2015 13:45:52 -0500 Content-Disposition: inline List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: guix-devel@gnu.org Hello, on my relatively slow ARM build machine with relatively fast storage (SSD), I notice that often there is an xz process taking 100% of CPU, while there is never more than 20MB/s written to disk. For instance, texlive-texmf takes a very long time to build and install into the store. Would it make sense to switch to a parallel (de-)compression tool to leverage higher numbers of cores? We have pbzip2 already in Guix, which is compatible with bzip2. As a negative point, we would increase the size of our packages and also the bandwidth requirement. So maybe this is not worth it, since we could also build more packages in parallel. Or are there parallel implementations of xz? Andreas