more package transformations: overrides

more package transformations: overrides

[Ricardo Wurmus]

Hi Guix,

when using package transformations the command line can become crowded
very quickly.  Sometimes I would like to be able to provide a file
containing a set of alternative packages that are used recursively in
the package dependency graph, as some sort of override.

What do you think about extending package transformations to allow for
reading override packages from a file?  The replacements would happen by
matching package names, so a package named “python-pytorch” in my
overrides file would replace the original “python-pytorch” package in
the input graph.  All these packages would be applied as overrides
together.

-- 
Ricardo

Re: more package transformations: overrides

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> when using package transformations the command line can become crowded
> very quickly.  Sometimes I would like to be able to provide a file
> containing a set of alternative packages that are used recursively in
> the package dependency graph, as some sort of override.

Seems like a good idea!  Cc'ing Sarthak as they might be also interested
in implementing this, as a good first experience with package
transformations.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: more package transformations: overrides

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> when using package transformations the command line can become crowded
> very quickly.  Sometimes I would like to be able to provide a file
> containing a set of alternative packages that are used recursively in
> the package dependency graph, as some sort of override.

The long-command-line issue can be addressed by using a manifest,
possibly generated (info "(guix) Writing Manifests").

> What do you think about extending package transformations to allow for
> reading override packages from a file?  The replacements would happen by
> matching package names, so a package named “python-pytorch” in my
> overrides file would replace the original “python-pytorch” package in
> the input graph.  All these packages would be applied as overrides
> together.

Could this override mechanism be itself a package transformation option?
:-)

  guix shell python-pytorch \
    --with-overrides=$HOME/.config/guix/my-stuff.scm

Perhaps ‘--with-overrides’ without any argument would pick a default
file name.

It would be implemented in terms of ‘package-input-rewriting/spec’ I
suppose.

Ludo’.

Re: more package transformations: overrides

[Maxim Cournoyer]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> when using package transformations the command line can become crowded
> very quickly.  Sometimes I would like to be able to provide a file
> containing a set of alternative packages that are used recursively in
> the package dependency graph, as some sort of override.
>
> What do you think about extending package transformations to allow for
> reading override packages from a file?  The replacements would happen by
> matching package names, so a package named “python-pytorch” in my
> overrides file would replace the original “python-pytorch” package in
> the input graph.  All these packages would be applied as overrides
> together.

Would you not be better off using a manifest in this case, using the
Guix API to apply the transformations?

-- 
Thanks,
Maxim

Re: more package transformations: overrides

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Ricardo,
>
> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Hi Guix,
>>
>> when using package transformations the command line can become crowded
>> very quickly.  Sometimes I would like to be able to provide a file
>> containing a set of alternative packages that are used recursively in
>> the package dependency graph, as some sort of override.
>>
>> What do you think about extending package transformations to allow for
>> reading override packages from a file?  The replacements would happen by
>> matching package names, so a package named “python-pytorch” in my
>> overrides file would replace the original “python-pytorch” package in
>> the input graph.  All these packages would be applied as overrides
>> together.
>
> Would you not be better off using a manifest in this case, using the
> Guix API to apply the transformations?

I do do that, but I find that this is quite a big jump for many people.
We have people who know how to add channels, or build a small custom
library by use the importer for simple packages, but who shy away from
using transformations in code.

Considering that the transformations API assumes that you understand
that higher order functions exist and what to do with them, I think it
would be good to offer a slightly less advanced method of overriding
packages.

Manifests can do anything, but this also makes them rather intimidating
for many of my colleagues.

-- 
Ricardo

Re: more package transformations: overrides

[Maxim Cournoyer]

Hi!

Ricardo Wurmus <rekado@elephly.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi Ricardo,
>>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>> Hi Guix,
>>>
>>> when using package transformations the command line can become crowded
>>> very quickly.  Sometimes I would like to be able to provide a file
>>> containing a set of alternative packages that are used recursively in
>>> the package dependency graph, as some sort of override.
>>>
>>> What do you think about extending package transformations to allow for
>>> reading override packages from a file?  The replacements would happen by
>>> matching package names, so a package named “python-pytorch” in my
>>> overrides file would replace the original “python-pytorch” package in
>>> the input graph.  All these packages would be applied as overrides
>>> together.
>>
>> Would you not be better off using a manifest in this case, using the
>> Guix API to apply the transformations?
>
> I do do that, but I find that this is quite a big jump for many people.
> We have people who know how to add channels, or build a small custom
> library by use the importer for simple packages, but who shy away from
> using transformations in code.
>
> Considering that the transformations API assumes that you understand
> that higher order functions exist and what to do with them, I think it
> would be good to offer a slightly less advanced method of overriding
> packages.
>
> Manifests can do anything, but this also makes them rather intimidating
> for many of my colleagues.

OK; then my question would be: why stop at transformations?  Perhaps we
could simply have a way to feed arbitrary guix command line arguments
from a file, e.g.

--8<---------------cut here---------------start------------->8---
guix shell --args-file=my-arguments.txt
--8<---------------cut here---------------end--------------->8---

where my-arguments.txt could contain anything, e.g.
--8<---------------cut here---------------start------------->8---
-CN
--with-source=package-name=some-url
[...]
--8<---------------cut here---------------end--------------->8---

Would something like that help?

-- 
Thanks,
Maxim

Re: more package transformations: overrides

[Ricardo Wurmus]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> Considering that the transformations API assumes that you understand
>> that higher order functions exist and what to do with them, I think it
>> would be good to offer a slightly less advanced method of overriding
>> packages.
>>
>> Manifests can do anything, but this also makes them rather intimidating
>> for many of my colleagues.
>
> OK; then my question would be: why stop at transformations?  Perhaps we
> could simply have a way to feed arbitrary guix command line arguments
> from a file, e.g.
>
> guix shell --args-file=my-arguments.txt

The transformations command line options are rather verbose:

  --with-input=python-pytorch=python-pytorch-with-extra-features

I think it would improve usability to be able to use the *same* names in
an override file.  Got a package with name “python-pytorch” in that
file?  It replaces “python-pytorch” in the graph.

~~

On the separate topic of reading arguments from elsewhere, though, I
think being able to read arguments from stdin could be useful.  Without
having to add a new option one could then do

    guix shell < my-arguments.txt

-- 
Ricardo

Re: more package transformations: overrides

[Sarthak Shah]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

Hello!
Thanks for the CC, Josselin!
I like Maxim's idea of providing all kinds of arguments through a separate
file or stdin.
Here's a tiny proof-of-concept script implementing that:
-----guix-shell-with-args.sh----
#!/usr/bin/env bash
argstr=""
while read line
do
  argstr="$argstr $line"
done < "${1:-/dev/stdin}"

guix shell $argstr
----------------------------------------
This takes arguments from either a file or stdin.
It is also possible to provide an arg file and supply additional arguments
from stdin (i.e. echo "arguments" | ./guix-shell-with-args.sh
/some/file-containing-args will take arguments from both)
This functionality could be baked into guix shell itself, wherein if you
pass an argument (say "-a") it will read arguments from either a supplied
file or stdin.
(of course, some error handling needs to be added)
I would like to have a more lisp-y way of doing this, however, if I'm being
honest.
Perhaps we could add some code that goes over a list from stdin/files?
Input could potentially look like this:
'(C N (with-source pkgname src) (with-input a b))
which would give
guix shell -C -N --with-source=pkgname=src --with-input=a=b
Wdyt?

---
Happy hacking!
Sarthak

On Thu, Mar 23, 2023 at 8:53 PM Ricardo Wurmus <rekado@elephly.net> wrote:

>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> >> Considering that the transformations API assumes that you understand
> >> that higher order functions exist and what to do with them, I think it
> >> would be good to offer a slightly less advanced method of overriding
> >> packages.
> >>
> >> Manifests can do anything, but this also makes them rather intimidating
> >> for many of my colleagues.
> >
> > OK; then my question would be: why stop at transformations?  Perhaps we
> > could simply have a way to feed arbitrary guix command line arguments
> > from a file, e.g.
> >
> > guix shell --args-file=my-arguments.txt
>
> The transformations command line options are rather verbose:
>
>   --with-input=python-pytorch=python-pytorch-with-extra-features
>
> I think it would improve usability to be able to use the *same* names in
> an override file.  Got a package with name “python-pytorch” in that
> file?  It replaces “python-pytorch” in the graph.
>
> ~~
>
> On the separate topic of reading arguments from elsewhere, though, I
> think being able to read arguments from stdin could be useful.  Without
> having to add a new option one could then do
>
>     guix shell < my-arguments.txt
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 3397 bytes --]

Re: more package transformations: overrides

[Maxim Cournoyer]

Hi!

Sarthak Shah <shahsarthakw@gmail.com> writes:

> Hello!
> Thanks for the CC, Josselin!
> I like Maxim's idea of providing all kinds of arguments through a separate
> file or stdin.
> Here's a tiny proof-of-concept script implementing that:
> -----guix-shell-with-args.sh----
> #!/usr/bin/env bash
> argstr=""
> while read line
> do
>   argstr="$argstr $line"
> done < "${1:-/dev/stdin}"
>
> guix shell $argstr

By separate file or stdin your really mean stdin, correct?  As in:

--8<---------------cut here---------------start------------->8---
guix something < args.txt or
echo "$args" | guix something
--8<---------------cut here---------------end--------------->8---

One note: since this mechanism could be generally useful, it should be
globally available (like 'guix build' options), for any 'guix'
invocation.

> ----------------------------------------
> This takes arguments from either a file or stdin.
> It is also possible to provide an arg file and supply additional arguments
> from stdin (i.e. echo "arguments" | ./guix-shell-with-args.sh
> /some/file-containing-args will take arguments from both)
> This functionality could be baked into guix shell itself, wherein if you
> pass an argument (say "-a") it will read arguments from either a supplied
> file or stdin.
> (of course, some error handling needs to be added)
> I would like to have a more lisp-y way of doing this, however, if I'm being
> honest.
> Perhaps we could add some code that goes over a list from stdin/files?
> Input could potentially look like this:
> '(C N (with-source pkgname src) (with-input a b))
> which would give
> guix shell -C -N --with-source=pkgname=src --with-input=a=b
> Wdyt?

At this point, we have a real Scheme API we can use :-).  If there's a
problem with using that, we should work toward improving it.

-- 
Thanks,
Maxim

Re: more package transformations: overrides

[Sarthak Shah]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

Hello,

> By separate file or stdin your really mean stdin, correct?  As in:
>
> --8<---------------cut here---------------start------------->8---
> guix something < args.txt or
> echo "$args" | guix something
> --8<---------------cut here---------------end--------------->8---
>
> One note: since this mechanism could be generally useful, it should be
> globally available (like 'guix build' options), for any 'guix'
> invocation.
>
Actually, while using this shell script you could also pass a file directly
i.e all of these will work:
-----(shell)------
./guix-shell-with-args /path/to/some/file-with-args
./guix-shell-with-args < /some/other/file
echo "something from stdin" | ./guix-shell-with-args
-------------------
If needed we could modify the script to only accept a file when a flag like
--read-args-from-file is passed. It's a proof of concept, and I think some
more work would need to be put in for actually implementing it since there
are probably a whole class of possible bugs and errors that we have not
considered.
I agree that it could be beneficial to all guix subcommands; I can see
myself using it with `guix package` for example. In this case we could
perhaps have a subcommand `guix with-args <subcommand>` which lets you
invoke <subcommand> (install, package, build etc) with arguments from stdin
or (optionally) a file.

At this point, we have a real Scheme API we can use :-).  If there's a
> problem with using that, we should work toward improving it.
>

 Yes, on further deliberation I realized that I'm basically writing a DSL
atop manifests   ^_^"

If more people see a use for this suggestion, I could try writing a more
polished shell script for `guix with-args`

---
Happy Hacking!
Sarthak

[-- Attachment #2: Type: text/html, Size: 2477 bytes --]

Re: more package transformations: overrides

[Maxim Cournoyer]

Hello,

Sarthak Shah <shahsarthakw@gmail.com> writes:

> Hello,
>
>> By separate file or stdin your really mean stdin, correct?  As in:
>>
>> --8<---------------cut here---------------start------------->8---
>> guix something < args.txt or
>> echo "$args" | guix something
>> --8<---------------cut here---------------end--------------->8---
>>
>> One note: since this mechanism could be generally useful, it should be
>> globally available (like 'guix build' options), for any 'guix'
>> invocation.
>>
> Actually, while using this shell script you could also pass a file directly
> i.e all of these will work:
> -----(shell)------
> ./guix-shell-with-args /path/to/some/file-with-args
> ./guix-shell-with-args < /some/other/file
> echo "something from stdin" | ./guix-shell-with-args
> -------------------

Interesting, althoug I think accepting a file would be perhaps going too
far, not sure.  I think targetting an explicit arg and stdin would be
a good starting point.

-- 
Thanks,
Maxim

Re: more package transformations: overrides

[Maxim Cournoyer]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> Considering that the transformations API assumes that you understand
>>> that higher order functions exist and what to do with them, I think it
>>> would be good to offer a slightly less advanced method of overriding
>>> packages.
>>>
>>> Manifests can do anything, but this also makes them rather intimidating
>>> for many of my colleagues.
>>
>> OK; then my question would be: why stop at transformations?  Perhaps we
>> could simply have a way to feed arbitrary guix command line arguments
>> from a file, e.g.
>>
>> guix shell --args-file=my-arguments.txt
>
> The transformations command line options are rather verbose:
>
>   --with-input=python-pytorch=python-pytorch-with-extra-features
>
> I think it would improve usability to be able to use the *same* names in
> an override file.  Got a package with name “python-pytorch” in that
> file?  It replaces “python-pytorch” in the graph.

It's still a bit too abstract in my mind.  Could you please provide a
skeletton of what the overrides file would look like?

-- 
Thanks,
Maxim

Re: more package transformations: overrides

[Maxim Cournoyer]

Hello,

Ricardo Wurmus <rekado@elephly.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> I think it would improve usability to be able to use the *same* names in
>>> an override file.  Got a package with name “python-pytorch” in that
>>> file?  It replaces “python-pytorch” in the graph.
>>
>> It's still a bit too abstract in my mind.  Could you please provide a
>> skeletton of what the overrides file would look like?
>
> It would just like a file you’d pass to “guix build -f”, except that it
> wouldn’t need to evaluate to a single package.  It would contain a bunch
> of package definitions.
>
> This would replace the need to set GUIX_PACKAGE_PATH (or pass -L) to a
> directory containing a module, and set up package transformations for
> each package.
>
> It would also be lighter than requiring the use of a manifest, which can
> get rather complicated when more than a few packages are to be replaced
> in the graph.

I see.  I can see how that could be convenient, but I'm a bit wary of
introducing yet another way to rewrite inputs (we have the CLI and the
manifest already).  On the other hand, the stdin/--args-file trick could
be universally useful and be a way to leverage the CLI in a more
user/vcs-friendly way (the CLI should be easier to approach than writing
a package definition -- if it's not let's discuss how to improve it), so
perhaps we could start with implementing that?

-- 
Thanks,
Maxim