From: Ricardo Wurmus <rekado@elephly.net>
To: Liliana Marie Prikler <liliana.prikler@ist.tugraz.at>
Cc: gwl-devel@gnu.org
Subject: Re: Processing large amounts of files
Date: Thu, 21 Mar 2024 15:34:02 +0100 [thread overview]
Message-ID: <87plvjd4el.fsf@elephly.net> (raw)
In-Reply-To: <2010bdb88116d64da3650b06e58979518b2c7277.camel@ist.tugraz.at>
Hi Liliana,
[-guix-devel@gnu.org, +gwl-devel@gnu.org]
(forgot to actually send this message a few days ago)
thanks for the report!
> I have a somewhat unusual workflow that requires me to do a number of
> processes on numerous, but small input files. The original is a bit
> unwieldy and takes several minutes to compile, but I've managed to
> produce a more understandable and better performing example. Note,
> that after a certain number of inputs, I get the following error:
>
> info: .16 Loading workflow file `meow.gwl'...
> info: 2.80 Computing workflow `cat'...
> run: 12.96 Executing: /bin/sh -c /gnu/store/kmssbjcdcabg9fh4nxscwwpnlb4px30h-gwl-meow.scm …
> error: 13.01 Wrong type argument in position 1: #f
It is frustrating that there is no backtrace. Reliable error handling
is hard.
When running with "-l all" I see this:
info: .75 Computing workflow `cat'...
debug: 3.13 Computing script for process `meow'
guix: 3.13 Looking up package `bash-minimal'
guix: 3.13 Opening inferior Guix at `/gnu/store/pb1nkrn3sg6a1j6c4r5j2ahygkf4vkv9-profile'
guix: 4.27 Looking up package `guix'
debug: 4.45 Generating all scripts and their dependencies.
debug: 4.89 Generating all scripts and their dependencies.
run: 6.73 Executing: /bin/sh -c /gnu/store/5idhbvhrwj3p53kkz2vikdn1ypncwj84-gwl-meow.scm '((inputs "/tmp/meow/0" ...
process: 8.80 In execvp of /bin/sh: Argument list too long
error: 8.80 Wrong type argument in position 1: #f
This at least tells us that the last error here is due to sh refusing to run.
> For comparison:
> time cat /tmp/meow/{0..7769}
> […]
>
> real 0m0,144s
> user 0m0,049s
> sys 0m0,094s
>
> It takes GWL 6 times longer to compute the workflow than to create the
> inputs in Guile, and 600 times longer than to actually execute the
> shell command. I think there is room for improvement :)
Yeah, not good. Do you have any recommendations?
--
Ricardo
next parent reply other threads:[~2024-03-24 15:40 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <2010bdb88116d64da3650b06e58979518b2c7277.camel@ist.tugraz.at>
2024-03-21 14:34 ` Ricardo Wurmus [this message]
2024-03-25 7:42 ` Processing large amounts of files Liliana Marie Prikler
2024-03-25 9:25 ` Ricardo Wurmus
2024-03-25 10:42 ` Ricardo Wurmus
2024-03-21 15:03 ` Ricardo Wurmus
2024-03-21 15:33 ` Liliana Marie Prikler
2024-03-26 21:30 ` Ricardo Wurmus
2024-03-27 7:10 ` Liliana Marie Prikler
2024-03-27 9:58 ` Ricardo Wurmus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.guixwl.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87plvjd4el.fsf@elephly.net \
--to=rekado@elephly.net \
--cc=gwl-devel@gnu.org \
--cc=liliana.prikler@ist.tugraz.at \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).