How to reference external program used in shell-scripts?

How to reference external program used in shell-scripts?

[Hartmut Goebel]

[-- Attachment #1.1: Type: text/plain, Size: 1842 bytes --]

Hi,

when I started (nut never finished) to package dtx some time ago, I
already ask similar. No I'm packaging debops, and I'm questioning the
answer I got last time.

Assume some program, shell-script, whatever is calling an external program.
What is the correct way to reference this? Shall it become an absolute
path, or just the basename.

In the DEB/RPM world, the basename is used, and package "pkgA"'s
meta-data ensure that the required package "pkgB" is installed in an
sufficient version. When "pkgB" is updated, "pkgA" does not need to be
rebuild or updated, as long as "pkgA"'s requirements are still
satisfied. Any installed "pkgA" will now use the new version of "pkgB".

In Guix, I was told to do so, the absolute path is used. This will
ensure the required package "pkgB" is installed in exact the version
used when building "pkgA". When the "pkgB" is updated, "pkgA" need to be
rebuild and updated. too. If not updated, existing installations of
"pkgA" will still use the *old* version of "pkgB".

I understand that in Guix package dependencies are not based ion
meta-data, but are tracked based on references found in the files. I
also understand that in many cases it is desirable to know exactly which
version of pkgB was used when running pkgA.

Yet in many cases the exact version of pkgB does not matter, a minimum
version might be enough. And if the version of pkgB was new enough when
bringing the pkgA to Guix, the version does not even really matter. E.g.
if the external program is less, git, encfs, or which.

So: What is the official recommendations? Is this already been stated in
to manual?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

Hi Hartmut,

> Assume some program, shell-script, whatever is calling an external program.
> What is the correct way to reference this? Shall it become an absolute
> path, or just the basename.

this depends on what the user may reasonably expect from the script.  If
it’s a core feature of the tool then I would expect to be able to use it
without having to manually install other packages.

If it’s a rare or optional feature it could be fine to use the basename
and assume that the user has the tool installed.

Generally, I try to make sure that required tools are either propagated
or that references to tools are made to absolute file names.

There is no hard rule.

-- 
Ricardo

Re: How to reference external program used in shell-scripts?

[Hartmut Goebel]

Am 08.08.19 um 16:50 schrieb Ricardo Wurmus:
> Generally, I try to make sure that required tools are either propagated
> or that references to tools are made to absolute file names.

Does this make a difference in regard to updates as I described?

I would expect in both cases, if a required package is updated, this
package will be updated, too. Right?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> I would expect in both cases, if a required package is updated, this
> package will be updated, too. Right?

Whenever an input is changed the package will be rebuilt, because we
can’t know if the presence of a package will affect the build or not.

In the case of patching references the presence of the input *will*
affect the output (as a reference to the absolute file name will be
recorded).  In the case of propagated inputs it’s really the same,
expect that the package will also be installed into the target profile.

--
Ricardo

Re: How to reference external program used in shell-scripts?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

Hi Ricardo,

Am 09.08.19 um 10:54 schrieb Ricardo Wurmus:
> Whenever an input is changed the package will be rebuilt, because we
> can’t know if the presence of a package will affect the build or not.
>
> In the case of patching references the presence of the input *will*
> affect the output (as a reference to the absolute file name will be
> recorded).  In the case of propagated inputs it’s really the same,
> expect that the package will also be installed into the target profile.

My concerns are not about building, but about installing. A concrete
example:

  * Ansible is a Python program running ssh via a path to
    /gnu/store/…-openssh-8.0p1/bin/ssh
  * Mary installs ansible.
  * Now openssh shows a serious bug and Mary updates openssh using "guix
    -u openssh"

Obviously this will *not* update ansible, and ansible will still use the
old, vulnerable version of openssh.

OTOH, if ansible would run ssh via $PATH, ansible would pick up the new
version of openssh.

FWIW: some way to install openssh automatically along with ansible,
while not specifying a specific version of openssh to be used, thus if
openssh is updated (but ansible is not), ansible will pick up the new
version.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 2303 bytes --]

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

Hi Hartmut,

> Am 09.08.19 um 10:54 schrieb Ricardo Wurmus:
>> Whenever an input is changed the package will be rebuilt, because we
>> can’t know if the presence of a package will affect the build or not.
>>
>> In the case of patching references the presence of the input *will*
>> affect the output (as a reference to the absolute file name will be
>> recorded).  In the case of propagated inputs it’s really the same,
>> expect that the package will also be installed into the target profile.
>
> My concerns are not about building, but about installing. A concrete
> example:
>
>   * Ansible is a Python program running ssh via a path to
>     /gnu/store/…-openssh-8.0p1/bin/ssh
>   * Mary installs ansible.
>   * Now openssh shows a serious bug and Mary updates openssh using "guix
>     -u openssh"

This is generally not a good idea, because this problem doesn’t only
affect dependent executables but all libraries as well.  Selective
upgrading means that not-upgraded packages will generally not benefit
from fixes.  This can range from libraries for extra features to really
fundamental functionality in glibc.

One could say now that the way around this problem in general would be …
to use a single global namespace for all libraries and all applications
— and this is how we would lose most of the advantages of Guix.

--
Ricardo

Re: How to reference external program used in shell-scripts?

[Ludovic Courtès]

Hello Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> My concerns are not about building, but about installing. A concrete
> example:
>
>   * Ansible is a Python program running ssh via a path to
>     /gnu/store/…-openssh-8.0p1/bin/ssh
>   * Mary installs ansible.
>   * Now openssh shows a serious bug and Mary updates openssh using "guix
>     -u openssh"
>
> Obviously this will *not* update ansible, and ansible will still use the
> old, vulnerable version of openssh.
>
> OTOH, if ansible would run ssh via $PATH, ansible would pick up the new
> version of openssh.

The whole idea of functional software deployment is that it’s stateless:
you can tell that /gnu/store/…-ansible-1.2.3 will always behave the
same, no matter what other programs are available on your machine.

Introducing “dynamic binding” (e.g., looking up programs in $PATH) would
allow for faster security updates in the example you gave, but that
would be at the expense of that core property I described above.  It
would be a regression.

I think what we need in this case is (1) fast security updates, which is
what grafts help us achieve, and (2) documentation that clarifies what
the deployment model is, such that Mary would know that ‘ansible’ also
needs to be upgraded in the example above.

Ludo’.

Re: How to reference external program used in shell-scripts?

[P via Development of GNU Guix and the GNU System distribution.]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, September 3, 2019 1:01 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hello Hartmut,
>
> Hartmut Goebel h.goebel@crazy-compilers.com skribis:
>
> > My concerns are not about building, but about installing. A concrete
> > example:
> >
> > -   Ansible is a Python program running ssh via a path to
> >     /gnu/store/…-openssh-8.0p1/bin/ssh
> >
> > -   Mary installs ansible.
> > -   Now openssh shows a serious bug and Mary updates openssh using "guix
> >     -u openssh"
> >
> >
> > Obviously this will not update ansible, and ansible will still use the
> > old, vulnerable version of openssh.
> > OTOH, if ansible would run ssh via $PATH, ansible would pick up the new
> > version of openssh.
>
> The whole idea of functional software deployment is that it’s stateless:
> you can tell that /gnu/store/…-ansible-1.2.3 will always behave the
> same, no matter what other programs are available on your machine.
>
> Introducing “dynamic binding” (e.g., looking up programs in $PATH) would
> allow for faster security updates in the example you gave, but that
> would be at the expense of that core property I described above. It
> would be a regression.
>
> I think what we need in this case is (1) fast security updates, which is
> what grafts help us achieve, and (2) documentation that clarifies what
> the deployment model is, such that Mary would know that ‘ansible’ also
> needs to be upgraded in the example above.
>
> Ludo’.

What about the performance side?
Can we tell Guix that an input is runtime only? Or only needed in certain stages of the build? Or is that better accomplished by splitting a package?
For example, if a large package (let's say... written in Rust) uses Lua for a utility script at run time, but doesn't touch it at build time changing the Lua version should not result in a recompilation.

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

P via Development of GNU Guix and the GNU System distribution. <guix-devel@gnu.org> writes:

> Can we tell Guix that an input is runtime only? Or only needed in
> certain stages of the build?

We cannot and we cannot guarantee that the presence of a package during
the build isn’t going to affect the build outcome.

> For example, if a large package (let's
> say... written in Rust) uses Lua for a utility script at run time, but
> doesn't touch it at build time changing the Lua version should not
> result in a recompilation.

We cannot guarantee this.

--
Ricardo

Re: How to reference external program used in shell-scripts?

[Hartmut Goebel]

Am 03.09.19 um 15:01 schrieb Ludovic Courtès:
> The whole idea of functional software deployment is that it’s stateless:
> you can tell that /gnu/store/…-ansible-1.2.3 will always behave the
> same, no matter what other programs are available on your machine.
>
> Introducing “dynamic binding” (e.g., looking up programs in $PATH) would
> allow for faster security updates in the example you gave, but that
> would be at the expense of that core property I described above.  It
> would be a regression.
>
> I think what we need in this case is (1) fast security updates, which is
> what grafts help us achieve, and (2) documentation that clarifies what
> the deployment model is, such that Mary would know that ‘ansible’ also
> needs to be upgraded in the example above.

I understand this.

Anyway: IMHO missing "dynamic binding" is one of the major drawbacks of
functional deployment, as it requires updating (and esp. downloading)
much more packages compared to a rpm/deb based system.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 03.09.19 um 15:01 schrieb Ludovic Courtès:
>> The whole idea of functional software deployment is that it’s stateless:
>> you can tell that /gnu/store/…-ansible-1.2.3 will always behave the
>> same, no matter what other programs are available on your machine.
>>
>> Introducing “dynamic binding” (e.g., looking up programs in $PATH) would
>> allow for faster security updates in the example you gave, but that
>> would be at the expense of that core property I described above.  It
>> would be a regression.
>>
>> I think what we need in this case is (1) fast security updates, which is
>> what grafts help us achieve, and (2) documentation that clarifies what
>> the deployment model is, such that Mary would know that ‘ansible’ also
>> needs to be upgraded in the example above.
>
> I understand this.
>
> Anyway: IMHO missing "dynamic binding" is one of the major drawbacks of
> functional deployment, as it requires updating (and esp. downloading)
> much more packages compared to a rpm/deb based system.

At the same time static binding is also one of the major advantages as
deployments are stateless and thus much more predictable, reliable, and
inspectable.

--
Ricardo

Re: How to reference external program used in shell-scripts?

[Konrad Hinsen]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
>
>> Anyway: IMHO missing "dynamic binding" is one of the major drawbacks of
>> functional deployment, as it requires updating (and esp. downloading)
>> much more packages compared to a rpm/deb based system.
>
> At the same time static binding is also one of the major advantages as
> deployments are stateless and thus much more predictable, reliable, and
> inspectable.

Indeed. We are having the same dynamic vs. static debate at the systems
level that has been going on at the programming language level for a
while. And for both situations, we still have to find the right
equilibrium between the two approaches, which both have their good and
bad aspects.

At the systems level, the last decades have seen a strong emphasis on
dynamic binding, in particular with the dominance of dynamic linking and
shared libraries. This has lead us into dependency hell to which we have
found quick-hack solutions (coarse-grained packages such as Docker
containers) and then more thorough ones (Nix and Guix).

Now the challenge is to put the right dose of dynamic binding back into
mostly static systems, and in the right place. For Guix, I'd say the
right place is the individual profile, and in particular the user's
default profile, so /usr/bin/env looks like a good start. My current way
of doing my work with Guix is to play dynamically in my default profile
but deploy stable applications in –pure environments. That's a bit more
work than what I could dream of, but it's sooooo much better than
everything else I have used before that I am not complaining.

Cheers,
  Konrad.