Re: GWL pipelined process composition ?

[Pjotr Prins <pjotr.public12@thebird.nl>]

On Wed, Jul 18, 2018 at 11:55:25PM +0200, zimoun wrote:
> Hi Roel,
> 
> Thank you for all your comments.
> 
> > Maybe we can come up with a convenient way to combine two processes
> > using a shell pipe.  But this needs more thought!
> 
> Yes, from my point of view, the classic shell pipe `|` has two strong
> limitations for workflows:
>  1. it does not compose at the 'process' level but at the procedure 'level'
>  2. it cannot deal with two inputs.
> 
> As an illustration for the point 1., it appears to me more "functional
> spirit" to write one process/task/unit corresponding to "samtools
> view" and another one about compressing "gzip -c". Then, if you have a
> process that filters some fastq, you can easily reuse the compress
> process, and composes it. For more complicated workflows, such as
> RNAseq or other, the composition seems an advantage.

Yes, but the original question was whether you could stream data
without writing to disk, right? Unix pipes are the system way of
providing that functionality - with the added advantage of parallel
processing between Unix processes. The downside, as you say is that it
is not that composable.

To make it composable you'd have to manage process communication -
using some network/socket protocol - and GWL would have to fire up the
processes in parallel so they can communicate - preferably on one box. 

That is a significant and fragile piece of functionality to add ;).

Error/failure handling in particular will be hard.

Unsurprisingly there are no systems that handle that well - that I am
aware. The best you'll get today is composable containers that 'talk'
with each other. But that is ad hoc network programming.

The great thing about the GWL is that it describes pipelines
deterministically and makes great use of GNU Guix. I think those are
killer features. 

Adding composable pipes will magnify the size of the code base and
make it fragile at the same time. Besides, network transport layers
will add another possibility of IO bottle necks. It is a whole project
in itself ;)

Probably a good idea to keep it simple.

I'd stick with pipes when possible. All pipelines can be described as
a combination of sequential processing and scatter/gather processing:
there will always be inefficiencies. To address these you'll need to
rewrite the software tools you want to run (as we did with sambamba at
the time, to replace samtools). 

In FP you are working in one space, so it is much easier to compose
(functions).

Pj.