From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id UF6KEgtNdmRa8wAASxT56A (envelope-from ) for ; Tue, 30 May 2023 21:22:51 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id YKifEQtNdmS1ewEAG6o9tA (envelope-from ) for ; Tue, 30 May 2023 21:22:51 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3E3703EAB for ; Tue, 30 May 2023 21:22:50 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q44sP-0001PZ-HK; Tue, 30 May 2023 15:19:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q44sJ-0001JK-Ft for guix-devel@gnu.org; Tue, 30 May 2023 15:19:21 -0400 Received: from mx0.riseup.net ([198.252.153.6]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q44sH-00062i-7E; Tue, 30 May 2023 15:19:19 -0400 Received: from fews01-sea.riseup.net (fews01-sea-pn.riseup.net [10.0.1.109]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.riseup.net", Issuer "R3" (not verified)) by mx0.riseup.net (Postfix) with ESMTPS id 4QW2JK2xbPz9t3L; Tue, 30 May 2023 19:19:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1685474353; bh=iT3ZPL4s0syMI/Q+84cvJ5uIuw7Go1cf4nIkC/OqaCE=; h=References:From:To:Cc:Subject:Date:In-reply-to:From; b=fs8HH0VIAhGuoOXy5HmBUSYNAJ6Vy79p7Hka8dD0cQUTRYFPgqOi/CWx3yxsX1cUx nrWmRMe3D8y/ACEvcQXQsH+ctsWonlKeHmaYe/tBagsRm/OIxaqi3EjrGqkkGlSBdD ERPbpPeDc1KZyo44pAhbBi7byF0B364tPkrXzSrA= X-Riseup-User-ID: AF485F50491D7086D26FC4A1D4E0C6FCCE59E21F65C88E3323B6B6EA380EB206 Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews01-sea.riseup.net (Postfix) with ESMTPSA id 4QW2J54jq2zJpgG; Tue, 30 May 2023 19:19:01 +0000 (UTC) References: <875y9jzl9m.fsf@gnu.org> <874jot19fd.fsf_-_@gnu.org> <87fs7rvv5s.fsf_-_@gnu.org> <878rddooy4.fsf@gnu.org> <87r0r4uv4x.fsf@gmail.com> <87ttvzhxm9.fsf@gnu.org> <87cz2it3yk.fsf@gmail.com> From: Csepp To: Simon Tournier Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Andreas Enge , guix-devel@gnu.org Subject: Re: How many bytes do we add (closure of guix) when adding one new package? Date: Tue, 30 May 2023 21:10:07 +0200 In-reply-to: <87cz2it3yk.fsf@gmail.com> Message-ID: <87r0qxvd9q.fsf@riseup.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=198.252.153.6; envelope-from=raingloom@riseup.net; helo=mx0.riseup.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Seal: i=1; s=key1; d=yhetil.org; t=1685474571; a=rsa-sha256; cv=none; b=CbhVqi57D9uB3WYi+WFfV/nBILlmzwWLJ2gbcay73Djhe0HgQZ7Y4v0AtUB6erFk6Xe0Qg 9wXY8baK5jp+rOUngbB/7HZLIRdgNHXB4Ft8dyuL/CZz5JUhrKoM/nGi/GeaIp8UeIRy5x RvWTixwL9f/1TQ8vqj+KGTKdWahID8vXu0bF9rGtyIgWQYzW57XBqMI0PkVAFlDLdVHCxL L+bG1BMDOgWEcC1CWZsN7VkctBW/fEOzYapARqHyhar2WHxz5ptXbYKc5x6Qw3jK4XrM58 DTmmzjaN6ymgwENyNd+UZ1MYR9Jqts8ijkGR4EeqqiTOmVvo37H3S7K680kt9A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=fs8HH0VI; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1685474571; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=b8ogaAytSkfBm74+pZ6xEj/WUZsd5i4v1XrQI/0vrMc=; b=SskH6awGANHzmRrdeTTPrFy5XMNgHM1edcakJNwrEGtPu4C6gJOiYiZD05p3JUmslfeW9q IZqOUurUmNIyKbxnmmscnUnvgQlVadgVxXW1fc1VA4TVKM6oHU+cMmIcOhEER1TiuL9DZW e3mDEsvZExboM60RJ2T4J0Zh+fZf1p3ena/mAmbWu1mJTkdRE5Ar+G06wdLBFqCN5rLB6f VEXfZs9mLJqA3bP+f6sg9qiehsC5lOvpDFhMEm3FJgKbo5UO14Sh8KGPF0FDLTxZ7AuKhp 5SmDQw4OJcoz1Iy/wiXqf6ND4h+dFn5VUeZylberJvOtg9gsqt6HptmLQ/WOcg== X-Migadu-Scanner: scn1.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=fs8HH0VI; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.47 X-Spam-Score: -2.47 X-Migadu-Queue-Id: 3E3703EAB X-TUID: Tz0MznTZoJgZ Simon Tournier writes: > Hi, > > On ven., 26 mai 2023 at 18:21, Ludovic Court=C3=A8s wrote: > >> I agree that .go files are quite big (.scm files as well, but we=E2=80= =99ve >> improved information density somewhat by removing input labels :-)). >> >> The size of .go files went down when we switch to the baseline compiler >> (aka. -O1): >> >> https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00071.html >> >> That thread has ideas of things to do to further reduce .go size. > > Just to put a figure on what means =E2=80=9Cbig=E2=80=9D: currently the .= go files are 5 > times bigger than their associated .scm. > > Somehow, it=E2=80=99s the trap of DSL. :-) Packages are declarative and t= he > information they declare is not dense. However, because they are > bytecompiled to a general programming language, their specificity is not > exploited. In an ideal world, the compiled binary representation of the > packages should be smaller than their human-readable text-file > counterpart. > > The mentioned improvement is nice. And it=E2=80=99s visible: > > --8<---------------cut here---------------start------------->8--- > 145M /gnu/store/nqrb3g4l59wd74w8mr9v0b992bj2sd1w-guix-d62c9b267-modules/l= ib/guile/3.0/site-ccache/gnu > 117M /gnu/store/s6rqlhqr750k44ynkqqj5mwjj2cs2yln-guix-a09968565-modules/l= ib/guile/3.0/site-ccache/gnu > 127M /gnu/store/ndii4bpyzh2rc05ya61s89rig9hdrl4k-guix-a0178d34f-modules/l= ib/guile/3.0/site-ccache/gnu > 164M /gnu/store/ni63a203jf61dwxlv8kr9b8x3vb1pdsp-guix-8e2f32cee-modules/l= ib/guile/3.0/site-ccache/gnu > --8<---------------cut here---------------end--------------->8--- > > However, it has almost no impact on the whole size; scaled by the number > of packages. > >> Download size has to be treated separately though. For example, =E2=80= =98git >> pull=E2=80=99 doesn=E2=80=99t redownload all of the repo or directory, a= nd it uses >> compression heavily. Thus, a few hundred bytes of additional .scm text >> translate in less than that. >> >> As for the rest, download size can be reduced for example by choosing a >> content-address transport, like something based on ERIS. >> >> I think we must look precisely at what we want to optimize=E2=80=94on-di= sk size, >> or bandwidth requirement, in particular=E2=80=94and look at the whole so= lution >> space. > > I think one direction is to tackle the way *package-modules* is built. > Because of that, Guix is building too much and the design is not optimal > =E2=80=93 whatever technical solutions we implement for improving after t= hat. > > On my poor laptop, Guix is becoming unusable because many operations are > becoming so slow =E2=80=93 when it=E2=80=99s still acceptable with APT of= Debian. For > instance, it=E2=80=99s something like 20 minutes for running =E2=80=9Cgui= x pull=E2=80=9D without > substitutes. And when I am traveling without a fast Internet > connection, it=E2=80=99s often too much for the network at hand. > > Currently, =E2=80=9Cguix pull=E2=80=9D is either building too much and do= wnloading too > much; by design. > > > Cheers, > simon Something I've been considering is if Guix could make use of database optimizations on its packages. Having access to Scheme for everything is nice, but using it as a storage solution is kind of silly when we are mostly just storing structs. Some kind of struct-of-arrays optimization could definitely reduce their size by a lot, might even speed up some operations. It makes zero sense to load full package definitions from disk for most queries, such as guix search, with an SoA representation we could load only the fields that we care about. ps.: Now I'm even more glad that I'm using a file system with transparent compression on all my Guix systems.