From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:33457) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hNl0q-0007Hi-PK for guix-patches@gnu.org; Mon, 06 May 2019 17:19:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hNl0o-0002ds-Vp for guix-patches@gnu.org; Mon, 06 May 2019 17:19:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44069) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hNl0o-0002df-Rm for guix-patches@gnu.org; Mon, 06 May 2019 17:19:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hNl0o-0002Rj-GG for guix-patches@gnu.org; Mon, 06 May 2019 17:19:02 -0400 Subject: [bug#34807] [PATCH 1/2] Add (guix lzlib). Resent-Message-ID: From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190310180209.11578-1-mail@ambrevar.xyz> <8736ne3855.fsf@gnu.org> <87pnp2f7gr.fsf@ambrevar.xyz> <8736lxdxn6.fsf@ambrevar.xyz> <87lfzm7fdz.fsf@gnu.org> <878svm5xic.fsf@ambrevar.xyz> <87ef5e0vvv.fsf@gnu.org> <87o94h527a.fsf@ambrevar.xyz> Date: Mon, 06 May 2019 23:18:38 +0200 In-Reply-To: <87o94h527a.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 04 May 2019 23:39:37 +0200") Message-ID: <87a7fzffip.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Pierre Neidhardt Cc: 34807@debbugs.gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Pierre, Pierre Neidhardt skribis: > It's definitely the ideal route. Something like guile-compress or > guile-archive, with a high-level abstraction for a collection of > bindings including zlib and lzlib for now. > > Sadly I don't have the time for it at the moment. Unless you do (:p) I > suggest we add a TODO item and keep it for later. Sounds good! Below are the Autoconf-related changes I made. Committed! We=E2=80=99ll take care of (guix self) when (guix lzlib) is actually used by other parts of the code. > Regarding guix publish and the farms, what shall we do? I think we should arrange for the client part, =E2=80=98guix substitute=E2= =80=99, to be ready to lzip-decode as soon as it talks to an lzip-capable server. Then we should add support in =E2=80=98guix publish=E2=80=99. At some late= r point, we=E2=80=99d deploy it on the build farms. For this migration to be incremental, we need (1) clients to be able to transparently switch to lzip when it=E2=80=99s available, and (2) servers t= o be able to produce both lzip archives (for new clients) and gzip archives (for old clients) during the transition period. That=E2=80=99s a bit of work in =E2=80=98guix publish=E2=80=99. It=E2=80= =99ll be extra CPU and storage usage on the build farm since during the transition period it=E2=80=99d hav= e to produce and store both gzip and lzip archives for each store item. I don=E2=80=99t really see any way around that, though. A difficulty is that narinfos currently include a fixed compression scheme: --8<---------------cut here---------------start------------->8--- $ wget -q -O - https://ci.guix.info/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh.narinf= o | head -3 StorePath: /gnu/store/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh-sed-4.5 URL: nar/gzip/nrkm1683p1cqnkcmhlmhiig9q9qd7xqh-sed-4.5 Compression: gzip --8<---------------cut here---------------end--------------->8--- So, depending on the client, =E2=80=98guix publish=E2=80=99 should return e= ither a narinfo-for-gzip or a narinfo-for-lzip. To make it possible, new clients could send an extra HTTP header, say =E2=80=98X-Guix-Compression=E2= =80=99, that would specify their preferred compression method(s). =E2=80=98guix publish= =E2=80=99 would take that into account when replying. How does that sound? Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable diff --git a/configure.ac b/configure.ac index ee7aa25742..3918550a79 100644 --- a/configure.ac +++ b/configure.ac @@ -251,14 +251,13 @@ AC_MSG_RESULT([$LIBZ]) AC_SUBST([LIBZ]) =20 dnl Library name of lzlib suitable for 'dynamic-link'. -GUIX_LIBLZ_LIBDIR([liblz_libdir]) -if test "x$liblz_libdir" =3D "x"; then +GUIX_LIBLZ_FILE_NAME([LIBLZ]) +if test "x$LIBLZ" =3D "x"; then LIBLZ=3D"liblz" else - LIBLZ=3D"$liblz_libdir/liblz" + # Strip the .so or .so.1 extension since that's what 'dynamic-link' expe= cts. + LIBLZ=3D"`echo $LIBLZ | sed -es'/\.so\(\.[[0-9.]]\+\)\?//g'`" fi -AC_MSG_CHECKING([for lzlib's shared library name]) -AC_MSG_RESULT([$LIBLZ]) AC_SUBST([LIBLZ]) =20 dnl Check for Guile-SSH, for the (guix ssh) module. diff --git a/guix/config.scm.in b/guix/config.scm.in index cd7036ca7e..0ada0f3c38 100644 --- a/guix/config.scm.in +++ b/guix/config.scm.in @@ -34,10 +34,10 @@ =20 %system %libz + %liblz %gzip %bzip2 - %xz - %liblz)) + %xz)) =20 ;;; Commentary: ;;; @@ -91,6 +91,9 @@ (define %libz "@LIBZ@") =20 +(define %liblz + "@LIBLZ@") + (define %gzip "@GZIP@") =20 @@ -100,9 +103,4 @@ (define %xz "@XZ@") =20 -(define %liblz - ;; TODO: Set this dynamically. - ;; "@LIBLZ@" - "/gnu/store/8db7vivi8p9mpkbphb8xy8gh2bkwc4iz-lzlib-1.11/lib/liblz") - ;;; config.scm ends here diff --git a/guix/self.scm b/guix/self.scm index 983f3514d3..74ea65240c 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -925,6 +925,7 @@ Info manual." %store-database-directory %config-directory %libz + ;; TODO: %liblz %gzip %bzip2 %xz)) @@ -971,8 +972,7 @@ Info manual." =20 (define %libz #+(and zlib - (file-append zlib "/lib/libz"))) - (define %liblz #f)) + (file-append zlib "/lib/libz")))) =20 ;; Guile 2.0 *requires* the 'define-module' to be at the ;; top-level or the 'toplevel-ref' in the resulting .go fil= e are diff --git a/m4/guix.m4 b/m4/guix.m4 index 78cc3777f8..d0c5ec0f08 100644 --- a/m4/guix.m4 +++ b/m4/guix.m4 @@ -1,5 +1,5 @@ dnl GNU Guix --- Functional package management for GNU -dnl Copyright =C2=A9 2012, 2013, 2014, 2015, 2016, 2018 Ludovic Court=C3= =A8s +dnl Copyright =C2=A9 2012, 2013, 2014, 2015, 2016, 2018, 2019 Ludovic Cour= t=C3=A8s dnl Copyright =C2=A9 2014 Mark H Weaver dnl Copyright =C2=A9 2017 Efraim Flashner dnl @@ -312,20 +312,18 @@ AC_DEFUN([GUIX_LIBZ_LIBDIR], [ $1=3D"$guix_cv_libz_libdir" ]) =20 -dnl GUIX_LIBLZ_LIBDIR VAR +dnl GUIX_LIBLZ_FILE_NAME VAR dnl -dnl Attempt to determine liblz's LIBDIR; store the result in VAR. -AC_DEFUN([GUIX_LIBLZ_LIBDIR], [ +dnl Attempt to determine liblz's absolute file name; store the result in V= AR. +AC_DEFUN([GUIX_LIBLZ_FILE_NAME], [ AC_REQUIRE([PKG_PROG_PKG_CONFIG]) - AC_CACHE_CHECK([lzlib's library directory], + AC_CACHE_CHECK([lzlib's file name], [guix_cv_liblz_libdir], - [guix_cv_liblz_libdir=3D"`$PKG_CONFIG lzlib --variable=3Dlibdir 2> /de= v/null`"]) - dnl TODO: lzlib has no pkg-config so we need the following trick to fi= nd its directory. - dnl old_LIBS=3D"$LIBS" - dnl LIBS=3D"-llz" - dnl AC_LINK_IFELSE([LZ_decompress_open();], - dnl [guix_cv_libz_libdir=3D"`ldd conftest$EXEEXT | grep liblz | sed = '-es/.*=3D> \([^ ]*\).*$/\1/g'`"]) - dnl LIBS=3D"$old_LIBS" + [old_LIBS=3D"$LIBS" + LIBS=3D"-llz" + AC_LINK_IFELSE([AC_LANG_SOURCE([int main () { return LZ_decompress_op= en(); }])], + [guix_cv_liblz_libdir=3D"`ldd conftest$EXEEXT | grep liblz | sed '-= es/.*=3D> \(.*\) .*$/\1/g'`"]) + LIBS=3D"$old_LIBS"]) $1=3D"$guix_cv_liblz_libdir" ]) =20 diff --git a/tests/lzlib.scm b/tests/lzlib.scm index a6631ce91c..cf53a9417d 100644 --- a/tests/lzlib.scm +++ b/tests/lzlib.scm @@ -26,8 +26,11 @@ =20 ;; Test the (guix lzlib) module. =20 -(unless (lzlib-available?) - (exit 77)) +(define-syntax-rule (test-assert* description exp) + (begin + (unless (lzlib-available?) + (test-skip 1)) + (test-assert description exp))) =20 (test-begin "lzlib") =20 @@ -68,41 +71,41 @@ (port-closed? parent) (bytevector=3D? received data))))))))))) =20 -(test-assert "null bytevector" +(test-assert* "null bytevector" (compress-and-decompress (make-bytevector (+ (random 100000) (* 20 1024))))) =20 -(test-assert "random bytevector" +(test-assert* "random bytevector" (compress-and-decompress (random-bytevector (+ (random 100000) (* 20 1024))))) -(test-assert "small bytevector" +(test-assert* "small bytevector" (compress-and-decompress (random-bytevector 127))) =20 -(test-assert "1 bytevector" +(test-assert* "1 bytevector" (compress-and-decompress (random-bytevector 1))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (2 * di= ctionary)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (2 * d= ictionary)" (compress-and-decompress (random-bytevector (* 2 (car (car (assoc-ref (@@ (guix lzlib) %compression-levels) (@@ (guix lzlib) %default-compression-level)= ))))))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB= )" (compress-and-decompress (random-bytevector (* 64 1024)))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB-= 1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB= -1)" (compress-and-decompress (random-bytevector (1- (* 64 1024))))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (64KiB+= 1)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (64KiB= +1)" (compress-and-decompress (random-bytevector (1+ (* 64 1024))))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB)" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB)" (compress-and-decompress (random-bytevector (* 1024 1024)))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB-1= )" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB-= 1)" (compress-and-decompress (random-bytevector (1- (* 1024 1024))))) =20 -(test-assert "Bytevector of size relative to Lzip internal buffers (1MiB+1= )" +(test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB+= 1)" (compress-and-decompress (random-bytevector (1+ (* 1024 1024))))) =20 (test-end) --=-=-=--