variable interpolation in code snippets

variable interpolation in code snippets

[Ricardo Wurmus]

Hi,

code snippets can now reference variables.  Here’s an example:

--8<---------------cut here---------------start------------->8---
define-module : snippets-wisp

import
  gwl processes
  gwl workflows
  gwl sugar

process: python-test
  package-inputs : list "python2"
  data-inputs
    list "sample.bam" "hg38.fa" "abc"
  procedure # python {
import os

def hello():
  print "hello from python 2"
  print "{{data-inputs}}"
  print "{{name}}"
  print os.environ["_GWL_PROCESS_DATA_INPUTS"]
  print os.environ["_GWL_PROCESS_NAME"]

hello()
}
--8<---------------cut here---------------end--------------->8---

The “python-test” process refers to the variables “data-inputs” and
“name”.  These variables are provided by the “process” record (values
for all defined fields are let bound, so that other fields can access
them), but you could access any other variable that’s in scope.

This is accomplished with the same reader macro that implements the “# ”
syntax for procedures.

The above example also shows access of the same values through
environment variables.  Note that in both cases all values are forced to
be strings.  You can’t have a Scheme list and use it natively as Python
array.  This whole “# ” syntax only operates on strings.  (You can have
a Scheme string that *looks* like Python syntax, of course, and splice
it into your inline Python code, but … you probably shouldn’t do that.)

--
Ricardo

Re: variable interpolation in code snippets

[zimoun]

Hi,

Wow!
It improves a lot the readibilty, IMHO.


On Wed, 20 Feb 2019 at 18:04, Ricardo Wurmus <rekado@elephly.net> wrote:
>
> process: python-test
>   package-inputs : list "python2"
>   data-inputs
>     list "sample.bam" "hg38.fa" "abc"
>   procedure # python {
> import os
>
> def hello():
>   print "hello from python 2"
>   print "{{data-inputs}}"

A quick question. The `data-inputs` is a list, so `print
"{{data-inputs}}` should return:
     "sample.bam" "hg38.fa" "abc"
Right?

How to refer to "hg38.fa" only?

   print "{{second data-inputs}}"



Thank you for this nice improvement !


All the best,
simon

Re: variable interpolation in code snippets

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> A quick question. The `data-inputs` is a list, so `print
> "{{data-inputs}}` should return:
>      "sample.bam" "hg38.fa" "abc"
> Right?
>
> How to refer to "hg38.fa" only?
>
>    print "{{second data-inputs}}"

That’s not possible, unfortunately.  Currently, the list is merely
converted to a string, so (list "sample.bam" "hg38.fa" "abc") becomes
the string “sample.bam hg38.fa abc”.  For singleton lists that’s exactly
what you want, but it’s not great for longer lists.

I thought about adding support for selectors, but I don’t know if I
should.  Maybe it would be better to allow for named inputs instead,
which would result in process-local bindings.

This would probably require changes to the records macro that we took
from Guix.  I’d prefer that over implementing a small language in a
reader macro.

--
Ricardo

Re: variable interpolation in code snippets

[zimoun]

On Tue, 26 Feb 2019 at 09:12, Ricardo Wurmus <rekado@elephly.net> wrote:
>
> I thought about adding support for selectors, but I don’t know if I
> should.  Maybe it would be better to allow for named inputs instead,
> which would result in process-local bindings.

`data-inputs` is often a list, say the genome (genome.fa) and the
sequences (R1.fq and R2.fq); then the procedure uses each as e.g.
    my-tool -i genome.fa -1 R1.fq -2 R2.fq

Personnally, I find the Snakemake notation clear:
  input:
     genome = genome.fa
     seqA = R1.fq
     seqB = R2.fq
  shell:
    my-tool -i {input.genome} -1 {input.seqA} -2 {input.seqB}

Well, I do not know which will be the best design.
process-local bindings with a "big let"?
procedure-local binding with let?
read macro with which "syntax"?
etc.

> This would probably require changes to the records macro that we took
> from Guix.  I’d prefer that over implementing a small language in a
> reader macro.

Hum? ok...
I am not sure to see what should be the final result?


In any case, the string interpolation already improves a lot compared
to the string-append. :-)



All the best
--
simon

Re: variable interpolation in code snippets

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Tue, 26 Feb 2019 at 09:12, Ricardo Wurmus <rekado@elephly.net> wrote:
>>
>> I thought about adding support for selectors, but I don’t know if I
>> should.  Maybe it would be better to allow for named inputs instead,
>> which would result in process-local bindings.
>
> `data-inputs` is often a list, say the genome (genome.fa) and the
> sequences (R1.fq and R2.fq); then the procedure uses each as e.g.
>     my-tool -i genome.fa -1 R1.fq -2 R2.fq
>
> Personnally, I find the Snakemake notation clear:
>   input:
>      genome = genome.fa
>      seqA = R1.fq
>      seqB = R2.fq
>   shell:
>     my-tool -i {input.genome} -1 {input.seqA} -2 {input.seqB}

Yes, I think this is pretty nice.  I’ll aim for something like this:

--8<---------------cut here---------------start------------->8---
process: foo
  data-inputs
    named-list
      foo = 1
      bar = hello
      baz = world
  procedure # bash { cat {{data-inputs.foo}} {{data-inputs.bar}} }
--8<---------------cut here---------------end--------------->8---

Haven’t thought much about it, but “named-list” (or whatever the final
name) would introduce a let binding or something.

(BTW: I don’t like that it’s called “data-inputs”. “inputs” is nicer and
fits well to “outputs”.)

-- 
Ricardo

Re: variable interpolation in code snippets

[zimoun]

On Tue, 26 Feb 2019 at 20:05, Ricardo Wurmus <rekado@elephly.net> wrote:
>
> Yes, I think this is pretty nice.  I’ll aim for something like this:
>
> --8<---------------cut here---------------start------------->8---
> process: foo
>   data-inputs
>     named-list
>       foo = 1
>       bar = hello
>       baz = world
>   procedure # bash { cat {{data-inputs.foo}} {{data-inputs.bar}} }
> --8<---------------cut here---------------end--------------->8---
>
> Haven’t thought much about it, but “named-list” (or whatever the final
> name) would introduce a let binding or something.

Yes!
The "named-list" would be the name of the reader macro, right?
(joke: "named-list" is a better name than "xyz-list" ;-)


> (BTW: I don’t like that it’s called “data-inputs”. “inputs” is nicer and
> fits well to “outputs”.)

Agree.
It was on my list asking for (bikeshedding) changes. ;-)

From my opinion, the change should be:
 - inputs -> packages
 - data-inputs -> inputs
 - outputs -> outputs


Cheers
--
simon

Re: variable interpolation in code snippets

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Tue, 26 Feb 2019 at 20:05, Ricardo Wurmus <rekado@elephly.net> wrote:
>>
>> Yes, I think this is pretty nice.  I’ll aim for something like this:
>>
>> --8<---------------cut here---------------start------------->8---
>> process: foo
>>   data-inputs
>>     named-list
>>       foo = 1
>>       bar = hello
>>       baz = world
>>   procedure # bash { cat {{data-inputs.foo}} {{data-inputs.bar}} }
>> --8<---------------cut here---------------end--------------->8---
>>
>> Haven’t thought much about it, but “named-list” (or whatever the final
>> name) would introduce a let binding or something.
>
> Yes!
> The "named-list" would be the name of the reader macro, right?
> (joke: "named-list" is a better name than "xyz-list" ;-)

I took a simpler route using keywords to tag or name items in a list.
This works now:

--8<---------------cut here---------------start------------->8---
process: foo
  data-inputs
    list
      #:foo 1
      #:bar hello
      #:baz world
  procedure # bash { cat {{data-inputs:foo}} {{data-inputs:bar}} }
--8<---------------cut here---------------end--------------->8---

You can choose to only tag one item if you want; it works just the same:

--8<---------------cut here---------------start------------->8---
process: foo
  data-inputs
    list
      1
      #:bar hello
      world
  procedure # bash { echo {{data-inputs:bar}} }
--8<---------------cut here---------------end--------------->8---


--
Ricardo

Re: variable interpolation in code snippets

[zimoun]

Hi Ricardo,

It improves the readibilty.
However, does the keyword `list` is mandatory ?
I find more readable something in this flavor:

--8<---------------cut here---------------start------------->8---
process: foo
  inputs:
      1
      #:bar hello
      world
  procedure # bash { echo {{inputs:bar}} }
--8<---------------cut here---------------end--------------->8---

With the renaming scheme that I proposed:
 - inputs -becomes-> packages
 - data-inputs -becomes-> inputs
 - outputs -becomes-> outputs

All the best,
simon

Re: variable interpolation in code snippets

[Ricardo Wurmus]

Hi simon,

> It improves the readibilty.
> However, does the keyword `list` is mandatory ?

Unfortunately it is mandatory.  Previously, I tried to give the record
field a “smart constructor” that takes either one value (a list or a let
binding resulting in a list) or — for convenience – multiple values that
are then turned into a list.

With the Guix-style records this does not seem to be possible.  If we
want to make this work we’d have to use our own extended records or
maybe switch to GOOPS.  GOOPS offers virtual slots that can have
slot-ref and slot-set! procedures, which would handle the conversion
transparently.  I think this would be a good way forward — and it would
decouple the GWL from the Guix version in use, because those extended
record are really made for Guix and may not forever match the needs of
the GWL.

> With the renaming scheme that I proposed:
>  - inputs -becomes-> packages
>  - data-inputs -becomes-> inputs
>  - outputs -becomes-> outputs

I still agree with these changes.  We’d only need to find a way to
support the old syntax for a while to allow for migrating existing
workflows (AFAIK that’s really just Roel’s workflows, but it better to
gradually deprecate the previous syntax).

--
Ricardo

Re: variable interpolation in code snippets

[Roel Janssen]

On Mon, 2019-06-03 at 18:04 +0200, Ricardo Wurmus wrote:
> Hi simon,
> 
> > It improves the readibilty.
> > However, does the keyword `list` is mandatory ?
> 
> Unfortunately it is mandatory.  Previously, I tried to give the record
> field a “smart constructor” that takes either one value (a list or a let
> binding resulting in a list) or — for convenience – multiple values that
> are then turned into a list.
> 
> With the Guix-style records this does not seem to be possible.  If we
> want to make this work we’d have to use our own extended records or
> maybe switch to GOOPS.  GOOPS offers virtual slots that can have
> slot-ref and slot-set! procedures, which would handle the conversion
> transparently.  I think this would be a good way forward — and it would
> decouple the GWL from the Guix version in use, because those extended
> record are really made for Guix and may not forever match the needs of
> the GWL.
> 
> > With the renaming scheme that I proposed:
> >  - inputs -becomes-> packages
> >  - data-inputs -becomes-> inputs
> >  - outputs -becomes-> outputs
> 
> I still agree with these changes.  We’d only need to find a way to
> support the old syntax for a while to allow for migrating existing
> workflows (AFAIK that’s really just Roel’s workflows, but it better to
> gradually deprecate the previous syntax).
> 

It's OK for me to deprecate/remove the previous syntax.

Kind regards,
Roel Janssen

Re: variable interpolation in code snippets

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> It improves the readibilty.
>> However, does the keyword `list` is mandatory ?
>
> Unfortunately it is mandatory.  Previously, I tried to give the record
> field a “smart constructor” that takes either one value (a list or a let
> binding resulting in a list) or — for convenience – multiple values that
> are then turned into a list.
>
> With the Guix-style records this does not seem to be possible.  If we
> want to make this work we’d have to use our own extended records or
> maybe switch to GOOPS.  GOOPS offers virtual slots that can have
> slot-ref and slot-set! procedures, which would handle the conversion
> transparently.  I think this would be a good way forward — and it would
> decouple the GWL from the Guix version in use, because those extended
> record are really made for Guix and may not forever match the needs of
> the GWL.

We’re now using GOOPS for the <process> type.  I implemented field
validation and implicit lists, so this is now possible:

--8<---------------cut here---------------start------------->8---
process: run-bash
  package-inputs "bash"
  data-inputs
    . "a"
    . #:awesome "b"
    . "c"
  procedure # bash {
    echo "The name of this process: {{name}}."
    echo "The most awesome of the data inputs is: {{data-inputs:awesome}}."
  }
--8<---------------cut here---------------end--------------->8---

The dots are necessary to continue broken lines, but it’s also fine to
put everything on one line:

   data-inputs "a" #:awesome "b" "c"

With a reader option we could shave off one more character from the
keyword syntax and use this instead:

   data-inputs "a" :awesome "b" "c"

Lists are still valid, of course, they are just optional:

   data-inputs : list "a" :awesome "b" "c"

or

   data-inputs
     list "a" :awesome "b" "c"

--
Ricardo