From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: =?utf-8?Q?=E2=80=98guix_publish=E2=80=99?= now compresses archives Date: Tue, 19 Jul 2016 00:22:42 +0200 Message-ID: <878twytwlp.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33000) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPGw5-0000Hw-Md for guix-devel@gnu.org; Mon, 18 Jul 2016 18:22:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bPGw2-0001Yn-Hn for guix-devel@gnu.org; Mon, 18 Jul 2016 18:22:49 -0400 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60098) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPGw2-0001YS-FB for guix-devel@gnu.org; Mon, 18 Jul 2016 18:22:46 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:41768 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1bPGw0-0000C0-JI for guix-devel@gnu.org; Mon, 18 Jul 2016 18:22:45 -0400 List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel Hi! =E2=80=98guix publish --compression=E2=80=99 can now compress archives in g= zip format, using zlib, which makes it much more usable in real life. See commit 4a1fc562ae5eedf40f6ae4eabe30580b0983b8f6. =E2=80=98guix publish=E2=80=99 is cache-friendly: it uses /nar/gzip URLs fo= r gzipped archives, and /nar for uncompressed archives. That way, even if =E2=80=98g= uix publish=E2=80=99 is restarted with different compression parameters, previo= usly announced /nar/* URLs remain valid. Please try it and report back. To recap, the difficulty with all this is that we couldn=E2=80=99t just cal= l out the =E2=80=98gzip=E2=80=99 command because =E2=80=98guix publish=E2=80=99 u= ses threads, and threads and fork(2) don=E2=80=99t go together well. So we needed zlib bindings, and wh= en discussing it earlier with Dave, I thought we=E2=80=99d have to wrap zlib= =E2=80=99s low-level API, which is painful, so we were sad and all. Next we had this hydra.gnu.org outage, which entailed more sadness. Then I realized that zlib has this high-level =E2=80=98gz=E2=80=99 API, whi= ch is easy and does what we need; hence, (guix zlib) provides bindings to that. The only limitation is that it needs a file descriptor as its source or sink and cannot compress into memory. That=E2=80=99s enough for =E2=80=98g= uix publish=E2=80=99 though. Comments welcome! Ludo=E2=80=99.