From: Ricardo Wurmus <rekado@elephly.net>
To: 55499@debbugs.gnu.org
Cc: "Ludovic Courtès" <ludo@gnu.org>
Subject: bug#55499: excessively large manifests due to propagation
Date: Wed, 18 May 2022 15:53:06 +0200 [thread overview]
Message-ID: <87sfp7kkim.fsf@elephly.net> (raw)
Packages of some languages rely heavily on propagation. R is one
example. Since the generated “manifest” file of a Guix profile records
entries for all propagated packages, this can get really big really
quickly.
A profile consisting only of four R packages (r-seurat, r-cistopic,
r-monocle3, and r-cicero-monocle3) results in a “manifest” file that
weighs 7.1MB. At the MDC I repeatedly encountered manifest files that
are exceeding 24MB.
Simply reading that big a file with
(call-with-input-file "huge-manifest" read)
takes several seconds. On the MDC cluster I observed a delay of about
27 seconds.
Disabling read positions with (read-disable 'positions) significantly
speeds this up (18s vs 27s), but it’s still very slow.
We may be able to speed things up by supporting definitions or
references in manifest files, so that we don’t need to repeat a sub-tree
when generating the file.
--
Ricardo
next reply other threads:[~2022-05-18 14:02 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-18 13:53 Ricardo Wurmus [this message]
2022-05-24 15:30 ` bug#55499: excessively large manifests due to propagation Ludovic Courtès
2022-05-25 4:35 ` Ricardo Wurmus
2022-05-31 16:09 ` bug#55499: [PATCH 0/3] Make 'manifest' files more compact Ludovic Courtès
2022-05-31 16:09 ` bug#55499: [PATCH 1/3] tests: Augment profile collision test Ludovic Courtès
2022-05-31 16:09 ` bug#55499: [PATCH 2/3] profiles: Do not repeat entries in 'manifest' file Ludovic Courtès
2022-05-31 17:35 ` Maxime Devos
2022-06-01 9:38 ` Ludovic Courtès
2022-05-31 16:09 ` bug#55499: [PATCH 3/3] squash! profiles: Make all entry fields optional Ludovic Courtès
2022-06-14 7:06 ` bug#55499: excessively large manifests due to propagation Ludovic Courtès
2022-07-01 21:54 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sfp7kkim.fsf@elephly.net \
--to=rekado@elephly.net \
--cc=55499@debbugs.gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).