Re: Google Summer of Code 2023 Inquiry

[Kyle <kyle@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

As a statistician who always wants to get the most information for the least effort, I am particularly interested in being able to reprioritize workflow jobs interactively within the equivalent portions of the topological sort. I thought perhaps this would be possible with GWL if it could talk to SLURM with DRMAA version 2 (https://en.wikipedia.org/wiki/DRMAA). This would also be more readily useful to researchers if Guix had a conveniently available slurm service which worked out of the box even on a single machine. 

Stepping back, there might be a more ambitious question hidden in there in terms of how to handle indeterminism in a deterministic workflow manager. Without that external information the problem just involves choosing your random seeds up front. However,  I would prefer to write a procedure which is constantly reprioritizing labeled sub jobs within their associated containers either until I hit a resource limit or I have achieved certain target statistical diagnostics. Perhaps I would want GWL to tell me how to replay my build after the fact so I can make that reproducible even though I didn't know what I needed to focus my computations on up front and let the computer do that. Making that sort of thing possible might be a longer term effort, but working out what's needed for initial steps might be a fun project.

On March 30, 2023 7:27:37 PM EDT, Spencer Skylar Chan <schan12@terpmail.umd.edu> wrote:
>Hi Ricardo,
>
>On 3/23/23 03:58, Ricardo Wurmus wrote:
>> Hi,
>> 
>> Spencer Skylar Chan <schan12@terpmail.umd.edu> writes:
>> 
>>> One approach could be to add CWL import/export capabilities to
>>> GWL. Then Snakemake/GWL conversion would be a 2 step process, using
>>> CWL as an intermediate step:
>>> 
>>> 1. Snakemake -> CWL
>>> 2. CWL -> GWL
>> 
>> This seems doable.
>
>Great! I've been reading the chapter in Evolutionary Genomics on different scalable workflows to understand this process better.
>
>>> However, CWL is not as expressive as Snakemake. There may be some
>>> details that are lost from Snakemake workflows.
>>> 
>>> So a 1-step Snakemake/GWL transpiler could be interesting, as both
>>> Snakemake/GWL use a domain-specific language inside a general purpose
>>> language (Python/Guile respectively). There may be a possibility to
>>> achieve more "accurate" translations between workflows.
>> 
>> Compared to the previous approach this seems vastly more complex.  It’s
>> one thing to *execute* Snakemake code without running it through Python,
>> but quite a bit more challenging to transpile Python to Scheme.
>> 
>> Personally, I wouldn’t know where to start.  Do you have an idea
>> already?
>> 
>
>Actually I was hoping you might have some ideas :)
>I do think that if the execution of the pipeline is more important than its representation (Snakemake or otherwise), then it would make more sense to focus efforts on increasing GWL's capabilities.
>
>Thanks,
>Skylar

[-- Attachment #2: Type: text/html, Size: 3803 bytes --]