unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Processing large amounts of files
@ 2024-03-13 11:22 Liliana Marie Prikler
  0 siblings, 0 replies; only message in thread
From: Liliana Marie Prikler @ 2024-03-13 11:22 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Hi Guix,

I have a somewhat unusual workflow that requires me to do a number of
processes on numerous, but small input files.  The original is a bit
unwieldy and takes several minutes to compile, but I've managed to
produce a more understandable and better performing example.  Note,
that after a certain number of inputs, I get the following error:

info: .16 Loading workflow file `meow.gwl'...
info: 2.80 Computing workflow `cat'...
run: 12.96 Executing: /bin/sh -c /gnu/store/kmssbjcdcabg9fh4nxscwwpnlb4px30h-gwl-meow.scm …
error: 13.01 Wrong type argument in position 1: #f

For comparison:
  time cat /tmp/meow/{0..7769}
  […]
  
  real	0m0,144s
  user	0m0,049s
  sys	0m0,094s

It takes GWL 6 times longer to compute the workflow than to create the
inputs in Guile, and 600 times longer than to actually execute the
shell command.  I think there is room for improvement :)

Cheers

[-- Attachment #2: meow.gwl --]
[-- Type: text/plain, Size: 431 bytes --]

use-modules
  guix build utils

define %directory "/tmp/meow"
mkdir-p %directory
define %files
  map
    lambda : i
      let
        : f : file %directory / : number->string i
        call-with-output-file f
          lambda : port
            display "meow\n" port
        . f
    iota 7769

process meow
  name "meow"
  inputs %files
  outputs "meow.cat"
  # {
    cat {{inputs}} > {{outputs}}
  }

workflow cat
  processes meow

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2024-03-13 11:24 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-13 11:22 Processing large amounts of files Liliana Marie Prikler

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).