Auto-create processe's output-path?

Auto-create processe's output-path?

[Olivier Dion via]

Hi,

I think it would be nice if GWL automatically creates the `output-path`
of a process.  

Currently, I have to do the following:
--8<---------------cut here---------------start------------->8---
output-path "output-of"
packages ... "coreutils" ...
# {
  ...
  mkdir {{output-path}}
  ...
}
--8<---------------cut here---------------end--------------->8---

I don't really like this because now I need "coretutils" as a dependency
-- which I tend to reduce to a minimal -- and I also need to manually
call `mkdir` in my script.

Would it be possible for GWL to do this instead?

-- 
Olivier Dion
oldiob.dev

Re: Auto-create processe's output-path?

[Ricardo Wurmus]

Olivier Dion via <gwl-devel@gnu.org> writes:

> Hi,
>
> I think it would be nice if GWL automatically creates the `output-path`
> of a process.  
>
> Currently, I have to do the following:
>
> output-path "output-of"
> packages ... "coreutils" ...
> # {
>   ...
>   mkdir {{output-path}}
>   ...
> }
>
> I don't really like this because now I need "coretutils" as a dependency
> -- which I tend to reduce to a minimal -- and I also need to manually
> call `mkdir` in my script.

I agree that using coreutils just to have mkdir only to create an output
location is inelegant.

That said: I don’t know if we should have OUTPUT-PATH at all.  It
doesn’t seem to do much other than setting the “out” environment
variable.  It seems to me that this is a vestigal remnant from simpler
times.

It is also used in PROCESS-OUTPUTS as a shared prefix for all declared
outputs.  Is this really useful enough to justify respecting it in
PROCESS-OUTPUTS?  Does it even work correctly with caching and
containerization?

(Maybe we should just remove it…?)

> Would it be possible for GWL to do this instead?

I guess we could have the GWL do this, but in some cases it may not be
desirable.  There are tools that get all offended when the parent
directory of their output files already exists.

-- 
Ricardo

Re: Auto-create processe's output-path?

[Olivier Dion via]

On Fri, 03 Jun 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> That said: I don’t know if we should have OUTPUT-PATH at all.  It
> doesn’t seem to do much other than setting the “out” environment
> variable.  It seems to me that this is a vestigal remnant from simpler
> times.
>
> It is also used in PROCESS-OUTPUTS as a shared prefix for all declared
> outputs.  Is this really useful enough to justify respecting it in
> PROCESS-OUTPUTS?  Does it even work correctly with caching and
> containerization?
>
> (Maybe we should just remove it…?)

I don't mind.  I use it avoid repeating the prefix but I think that a
concept of `working directory' would probably be better.  Maybe an
example of mine would explain better.

I have a pipeline of processes.  These are all templated and I generate
hundreds of these pipelines with different parameters.  Everything is
auto-connected with inputs/outputs matching.  What is important to me is
that all inputs/outputs of a pipeline are separated in different
directories.
 
Setting a working directory that is parameterizable for the processes
would do this cleanly and I believe would also work in a containerized
environment.  I'm not sure what would be the implication for the
caching, but I belive that it does not impact it.

>> Would it be possible for GWL to do this instead?
>
> I guess we could have the GWL do this, but in some cases it may not be
> desirable.  There are tools that get all offended when the parent
> directory of their output files already exists.

Agree.

-- 
Olivier Dion
oldiob.dev

Re: Auto-create processe's output-path?

[Ricardo Wurmus]

Olivier Dion <olivier.dion@polymtl.ca> writes:

> I have a pipeline of processes.  These are all templated and I generate
> hundreds of these pipelines with different parameters.  Everything is
> auto-connected with inputs/outputs matching.  What is important to me is
> that all inputs/outputs of a pipeline are separated in different
> directories.
>  
> Setting a working directory that is parameterizable for the processes
> would do this cleanly and I believe would also work in a containerized
> environment.

This sounds like a good idea.  Better than “output-path”, which is hard
to understand, conceptually collides with “outputs”, and is used as a
prefix.  Specifying a working directory sounds like an improvement.

Would you like to add a patch to that effect?

-- 
Ricardo

Re: Auto-create processe's output-path?

[Olivier Dion via]

On Mon, 06 Jun 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Olivier Dion <olivier.dion@polymtl.ca> writes:
>
>> I have a pipeline of processes.  These are all templated and I generate
>> hundreds of these pipelines with different parameters.  Everything is
>> auto-connected with inputs/outputs matching.  What is important to me is
>> that all inputs/outputs of a pipeline are separated in different
>> directories.
>>  
>> Setting a working directory that is parameterizable for the processes
>> would do this cleanly and I believe would also work in a containerized
>> environment.
>
> This sounds like a good idea.  Better than “output-path”, which is hard
> to understand, conceptually collides with “outputs”, and is used as a
> prefix.  Specifying a working directory sounds like an improvement.
>
> Would you like to add a patch to that effect?

Yes.  I'll send a patch soon!

-- 
Olivier Dion
oldiob.dev