Processing large amounts of files

Processing large amounts of files

[Liliana Marie Prikler]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Hi Guix,

I have a somewhat unusual workflow that requires me to do a number of
processes on numerous, but small input files.  The original is a bit
unwieldy and takes several minutes to compile, but I've managed to
produce a more understandable and better performing example.  Note,
that after a certain number of inputs, I get the following error:

info: .16 Loading workflow file `meow.gwl'...
info: 2.80 Computing workflow `cat'...
run: 12.96 Executing: /bin/sh -c /gnu/store/kmssbjcdcabg9fh4nxscwwpnlb4px30h-gwl-meow.scm …
error: 13.01 Wrong type argument in position 1: #f

For comparison:
  time cat /tmp/meow/{0..7769}
  […]
  
  real	0m0,144s
  user	0m0,049s
  sys	0m0,094s

It takes GWL 6 times longer to compute the workflow than to create the
inputs in Guile, and 600 times longer than to actually execute the
shell command.  I think there is room for improvement :)

Cheers

[-- Attachment #2: meow.gwl --]
[-- Type: text/plain, Size: 431 bytes --]

use-modules
  guix build utils

define %directory "/tmp/meow"
mkdir-p %directory
define %files
  map
    lambda : i
      let
        : f : file %directory / : number->string i
        call-with-output-file f
          lambda : port
            display "meow\n" port
        . f
    iota 7769

process meow
  name "meow"
  inputs %files
  outputs "meow.cat"
  # {
    cat {{inputs}} > {{outputs}}
  }

workflow cat
  processes meow