unofficial mirror of gwl-devel@gnu.org
 help / color / mirror / Atom feed
* Auto-create processe's output-path?
@ 2022-06-02 16:33 Olivier Dion via
  2022-06-03 18:43 ` Ricardo Wurmus
  0 siblings, 1 reply; 5+ messages in thread
From: Olivier Dion via @ 2022-06-02 16:33 UTC (permalink / raw)
  To: gwl-devel

Hi,

I think it would be nice if GWL automatically creates the `output-path`
of a process.  

Currently, I have to do the following:
--8<---------------cut here---------------start------------->8---
output-path "output-of"
packages ... "coreutils" ...
# {
  ...
  mkdir {{output-path}}
  ...
}
--8<---------------cut here---------------end--------------->8---

I don't really like this because now I need "coretutils" as a dependency
-- which I tend to reduce to a minimal -- and I also need to manually
call `mkdir` in my script.

Would it be possible for GWL to do this instead?

-- 
Olivier Dion
oldiob.dev



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Auto-create processe's output-path?
  2022-06-02 16:33 Auto-create processe's output-path? Olivier Dion via
@ 2022-06-03 18:43 ` Ricardo Wurmus
  2022-06-03 19:05   ` Olivier Dion via
  0 siblings, 1 reply; 5+ messages in thread
From: Ricardo Wurmus @ 2022-06-03 18:43 UTC (permalink / raw)
  To: Olivier Dion; +Cc: gwl-devel


Olivier Dion via <gwl-devel@gnu.org> writes:

> Hi,
>
> I think it would be nice if GWL automatically creates the `output-path`
> of a process.  
>
> Currently, I have to do the following:
>
> output-path "output-of"
> packages ... "coreutils" ...
> # {
>   ...
>   mkdir {{output-path}}
>   ...
> }
>
> I don't really like this because now I need "coretutils" as a dependency
> -- which I tend to reduce to a minimal -- and I also need to manually
> call `mkdir` in my script.

I agree that using coreutils just to have mkdir only to create an output
location is inelegant.

That said: I don’t know if we should have OUTPUT-PATH at all.  It
doesn’t seem to do much other than setting the “out” environment
variable.  It seems to me that this is a vestigal remnant from simpler
times.

It is also used in PROCESS-OUTPUTS as a shared prefix for all declared
outputs.  Is this really useful enough to justify respecting it in
PROCESS-OUTPUTS?  Does it even work correctly with caching and
containerization?

(Maybe we should just remove it…?)

> Would it be possible for GWL to do this instead?

I guess we could have the GWL do this, but in some cases it may not be
desirable.  There are tools that get all offended when the parent
directory of their output files already exists.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Auto-create processe's output-path?
  2022-06-03 18:43 ` Ricardo Wurmus
@ 2022-06-03 19:05   ` Olivier Dion via
  2022-06-06 10:26     ` Ricardo Wurmus
  0 siblings, 1 reply; 5+ messages in thread
From: Olivier Dion via @ 2022-06-03 19:05 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: gwl-devel

On Fri, 03 Jun 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> That said: I don’t know if we should have OUTPUT-PATH at all.  It
> doesn’t seem to do much other than setting the “out” environment
> variable.  It seems to me that this is a vestigal remnant from simpler
> times.
>
> It is also used in PROCESS-OUTPUTS as a shared prefix for all declared
> outputs.  Is this really useful enough to justify respecting it in
> PROCESS-OUTPUTS?  Does it even work correctly with caching and
> containerization?
>
> (Maybe we should just remove it…?)

I don't mind.  I use it avoid repeating the prefix but I think that a
concept of `working directory' would probably be better.  Maybe an
example of mine would explain better.

I have a pipeline of processes.  These are all templated and I generate
hundreds of these pipelines with different parameters.  Everything is
auto-connected with inputs/outputs matching.  What is important to me is
that all inputs/outputs of a pipeline are separated in different
directories.
 
Setting a working directory that is parameterizable for the processes
would do this cleanly and I believe would also work in a containerized
environment.  I'm not sure what would be the implication for the
caching, but I belive that it does not impact it.

>> Would it be possible for GWL to do this instead?
>
> I guess we could have the GWL do this, but in some cases it may not be
> desirable.  There are tools that get all offended when the parent
> directory of their output files already exists.

Agree.

-- 
Olivier Dion
oldiob.dev


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Auto-create processe's output-path?
  2022-06-03 19:05   ` Olivier Dion via
@ 2022-06-06 10:26     ` Ricardo Wurmus
  2022-06-06 13:26       ` Olivier Dion via
  0 siblings, 1 reply; 5+ messages in thread
From: Ricardo Wurmus @ 2022-06-06 10:26 UTC (permalink / raw)
  To: Olivier Dion; +Cc: gwl-devel


Olivier Dion <olivier.dion@polymtl.ca> writes:

> I have a pipeline of processes.  These are all templated and I generate
> hundreds of these pipelines with different parameters.  Everything is
> auto-connected with inputs/outputs matching.  What is important to me is
> that all inputs/outputs of a pipeline are separated in different
> directories.
>  
> Setting a working directory that is parameterizable for the processes
> would do this cleanly and I believe would also work in a containerized
> environment.

This sounds like a good idea.  Better than “output-path”, which is hard
to understand, conceptually collides with “outputs”, and is used as a
prefix.  Specifying a working directory sounds like an improvement.

Would you like to add a patch to that effect?

-- 
Ricardo


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Auto-create processe's output-path?
  2022-06-06 10:26     ` Ricardo Wurmus
@ 2022-06-06 13:26       ` Olivier Dion via
  0 siblings, 0 replies; 5+ messages in thread
From: Olivier Dion via @ 2022-06-06 13:26 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: gwl-devel

On Mon, 06 Jun 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Olivier Dion <olivier.dion@polymtl.ca> writes:
>
>> I have a pipeline of processes.  These are all templated and I generate
>> hundreds of these pipelines with different parameters.  Everything is
>> auto-connected with inputs/outputs matching.  What is important to me is
>> that all inputs/outputs of a pipeline are separated in different
>> directories.
>>  
>> Setting a working directory that is parameterizable for the processes
>> would do this cleanly and I believe would also work in a containerized
>> environment.
>
> This sounds like a good idea.  Better than “output-path”, which is hard
> to understand, conceptually collides with “outputs”, and is used as a
> prefix.  Specifying a working directory sounds like an improvement.
>
> Would you like to add a patch to that effect?

Yes.  I'll send a patch soon!

-- 
Olivier Dion
oldiob.dev


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2022-06-06 13:32 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-02 16:33 Auto-create processe's output-path? Olivier Dion via
2022-06-03 18:43 ` Ricardo Wurmus
2022-06-03 19:05   ` Olivier Dion via
2022-06-06 10:26     ` Ricardo Wurmus
2022-06-06 13:26       ` Olivier Dion via

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).