unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
From: Ricardo Wurmus <rekado@elephly.net>
To: 55499@debbugs.gnu.org
Cc: "Ludovic Courtès" <ludo@gnu.org>
Subject: bug#55499: excessively large manifests due to propagation
Date: Wed, 18 May 2022 15:53:06 +0200	[thread overview]
Message-ID: <87sfp7kkim.fsf@elephly.net> (raw)

Packages of some languages rely heavily on propagation.  R is one
example.  Since the generated “manifest” file of a Guix profile records
entries for all propagated packages, this can get really big really
quickly.

A profile consisting only of four R packages (r-seurat, r-cistopic,
r-monocle3, and r-cicero-monocle3) results in a “manifest” file that
weighs 7.1MB.  At the MDC I repeatedly encountered manifest files that
are exceeding 24MB.

Simply reading that big a file with

    (call-with-input-file "huge-manifest" read)

takes several seconds.  On the MDC cluster I observed a delay of about
27 seconds.

Disabling read positions with (read-disable 'positions) significantly
speeds this up (18s vs 27s), but it’s still very slow.

We may be able to speed things up by supporting definitions or
references in manifest files, so that we don’t need to repeat a sub-tree
when generating the file.

-- 
Ricardo




             reply	other threads:[~2022-05-18 14:02 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-18 13:53 Ricardo Wurmus [this message]
2022-05-24 15:30 ` bug#55499: excessively large manifests due to propagation Ludovic Courtès
2022-05-25  4:35   ` Ricardo Wurmus
2022-05-31 16:09 ` bug#55499: [PATCH 0/3] Make 'manifest' files more compact Ludovic Courtès
2022-05-31 16:09   ` bug#55499: [PATCH 1/3] tests: Augment profile collision test Ludovic Courtès
2022-05-31 16:09   ` bug#55499: [PATCH 2/3] profiles: Do not repeat entries in 'manifest' file Ludovic Courtès
2022-05-31 17:35     ` Maxime Devos
2022-06-01  9:38       ` Ludovic Courtès
2022-05-31 16:09   ` bug#55499: [PATCH 3/3] squash! profiles: Make all entry fields optional Ludovic Courtès
2022-06-14  7:06   ` bug#55499: excessively large manifests due to propagation Ludovic Courtès
2022-07-01 21:54     ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87sfp7kkim.fsf@elephly.net \
    --to=rekado@elephly.net \
    --cc=55499@debbugs.gnu.org \
    --cc=ludo@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).