From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: Re: Publishing with Lzip Date: Tue, 12 Mar 2019 14:19:35 +0100 Message-ID: <874l88fcwo.fsf@gnu.org> References: <87mum2zixw.fsf@ambrevar.xyz> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([209.51.188.92]:54721) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h3hTm-0002y9-2m for guix-devel@gnu.org; Tue, 12 Mar 2019 09:30:04 -0400 In-Reply-To: <87mum2zixw.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sun, 10 Mar 2019 19:24:27 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pierre Neidhardt Cc: guix-devel Hi Pierre, Pierre Neidhardt skribis: > I've just sent a patch of a first draft for the Lzip bindings: > > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D34807 > > Switching from Gzip (Zlib) to Lzip would save up to some 50% disk usage > for substitutes which would greatly benefit the servers and help > reducing the bandwidth usage. Woohoo, awesome! > A few things are missing in the patch and I would probably need some help: > > - The %liblz variable in guix/config.scm.in is not set dynamically by > autoconf. > This is because lzlib does not have a pkg-config entry. I don't know > much about autoconf, so if someone knows how to do this properly... I can give it a spin. > - I'm still a bit confused about how the API is supposed to work. There > is a lz-(de)compress-finish function which is supposed to be called > according to the examples but the manual does not really says why. If > I call it, the tests fail. > > - I'm also not 100% I'm doing the right thing with the encoder/decoder: > should we write everything first, then read as much as we can? Or > chain write-read calls like in bbexample.c? > > - I'm not 100% sure either that the terminating chunk will always be > compressed / decompressed. (It works in the test though.) I don't > really understand how Lzlib handles that part. It=E2=80=99d be great to gain some confidence about these things. :-) I ha= ven=E2=80=99t looked at the patch yet, but if you haven=E2=80=99t done it yet, I=E2=80=99= d suggest having tests like the one in tests/zlib.scm (compress and decompress a bytevector of a random size with random contents), and you could possibly perform more =E2=80=9Cstressful=E2=80=9D tests manually as well (t= ry to compress/decompress tarballs, etc.) I=E2=80=99d also recommend to re-read the API doc in the headers or whateve= r. IME these APIs are very tricky to use and one has to pay attention to every small detail. > - How can I map between C enums and Guile with dynamic FFI? This would > be useful to have improve error messages. According to the C standard an enum is an =E2=80=98int=E2=80=99. So mappin= g them is just a matter of producing/consuming ints. The values of the enum start from 0 and are incremented by 1 from then on, unless specific values are provided. > - lzlib.scm is not used for publishing in the patch. Will do that > later. What are the strategies for transitioning from .gz to .lz? I > suggest the following: > > - On publishing, replace .gz with .lz compression. > - When extracting, check the type and call the appropriate format decom= pressor. The strategy I think would be to first have the complete tool chain in Guix, that is support in =E2=80=98guix substitute=E2=80=99 and =E2=80=98gui= x publish=E2=80=99. We won=E2=80=99t be able to change our servers to produce lzip overnight, because old instances of =E2=80=98guix substitute=E2=80=99 wouldn=E2=80=99t= be able to consume it. Perhaps one option would be to allow =E2=80=98guix publish=E2=80=99 to prod= uce both gzip and lzip, which we=E2=80=99d use during some transition period. The diffic= ulty would be that narinfos currently provide just one URL for the nar, so we=E2=80=99d need to either provide several URLs, or provide the right URL = based on some appropriate HTTP request header. Let=E2=80=99s focus on the rest f= or now. :-) Thank you! Ludo=E2=80=99.