The case for moving raw binaries

The case for moving raw binaries

[Liliana Marie Prikler]

Hi Guix,

"raw binaries" (henceforth rawbins) are the unwrapped binaries that
Guix leaves behind in $PACKAGE/bin with the .$WRAPPER-real name.  This
practise causes several issues.  For one, those rawbins are visible in
the shell by typing a dot and using tab completion.  What's more, in
some build systems there might be two (or even more) off them.  This
makes a generic wrap after wrap pattern almost impossible to achieve.

So, what's the fix?  I propose moving rawbins to a different location.
libexec would spring to mind as a place in which we could hide them, so
would a new directory in the root of $PACKAGE.  Other than that, adding
a rawbin output would also be possible, but I am not certain whether
that'd be the right tradeoff.

So, what do you think?  Any candidates for $RAWBIN_DIR that we can
bikeshed?  Any disagreements?

Cheers

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

Liliana Marie Prikler schreef op do 28-04-2022 om 18:37 [+0200]:
> So, what do you think?  Any candidates for $RAWBIN_DIR that we can
> bikeshed?  Any disagreements?

/gnu/HASH-.../unwrapped-bin.  Benefit above /gnu/HASH-
.../libexec/SOMETHING/bin:  the 'dirname' of $0 points to PREXIX, like
for binaries in .../bin, whereas for libexec, the dirname points to a
subdirectory of PREFIX, which some applications might not expect.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

Liliana Marie Prikler schreef op do 28-04-2022 om 18:37 [+0200]:
> the shell by typing a dot and using tab completion.  What's more, in
> some build systems there might be two (or even more) off them.  This
> makes a generic wrap after wrap pattern almost impossible to achieve.
> 
> So, what's the fix?  I propose moving rawbins to a different location.
> libexec would spring to mind as a place in which we could hide them, so

How does this help with double wrapping?  Whether the wrappers /
originals are put in /bin or $RAWBIN_DIR, it's still wrapped twice.

Also, FWIW, double-wrapping works nicely for 'wrap-program' -- it just
appende or prepended some extra X=Y lines.  With some work and test
cases, 'wrap-script' could be extended to support such a thing as well.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Donnerstag, dem 28.04.2022 um 18:55 +0200 schrieb Maxime Devos:
> Liliana Marie Prikler schreef op do 28-04-2022 om 18:37 [+0200]:
> > the shell by typing a dot and using tab completion.  What's more,
> > in some build systems there might be two (or even more) off them. 
> > This makes a generic wrap after wrap pattern almost impossible to
> > achieve.
> > 
> > So, what's the fix?  I propose moving rawbins to a different
> > location.  libexec would spring to mind as a place in which we
> > could hide them, so
> 
> How does this help with double wrapping?  Whether the wrappers /
> originals are put in /bin or $RAWBIN_DIR, it's still wrapped twice.
Because $RAWBIN_DIR can be ignored when wrapping.  This means that
stuff that's already in it won't be added again.

> Also, FWIW, double-wrapping works nicely for 'wrap-program' -- it
> just appende or prepended some extra X=Y lines.  With some work and
> test cases, 'wrap-script' could be extended to support such a thing
> as well.
Constructing the wrapper is not so much the problem, it's not wrapping
the already wrapped binaries.  Plus the .-real binaries showing up in
$PATH remains an issue if they don't move :)

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

Liliana Marie Prikler schreef op do 28-04-2022 om 19:27 [+0200]:
> > How does this help with double wrapping?  Whether the wrappers /
> > originals are put in /bin or $RAWBIN_DIR, it's still wrapped twice.
> Because $RAWBIN_DIR can be ignored when wrapping.  This means that
> stuff that's already in it won't be added again.
> [...]
> Constructing the wrapper is not so much the problem, it's not
> wrapping the already wrapped binaries. 

Why can $RAWBIN_DIR be ignored when wrapping?  If the build system just
sets $X during its wrappers, but the application needs $Y as well
(wrapped in a build phase), then if the extra wrapping is cancelled,
then the application won't get its $Y variable, which seems like a bug
to me.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[raingloom]

On Thu, 28 Apr 2022 21:57:04 +0200
Maxime Devos <maximedevos@telenet.be> wrote:

> Liliana Marie Prikler schreef op do 28-04-2022 om 19:27 [+0200]:
> > > How does this help with double wrapping?  Whether the wrappers /
> > > originals are put in /bin or $RAWBIN_DIR, it's still wrapped
> > > twice.  
> > Because $RAWBIN_DIR can be ignored when wrapping.  This means that
> > stuff that's already in it won't be added again.
> > [...]
> > Constructing the wrapper is not so much the problem, it's not
> > wrapping the already wrapped binaries.   
> 
> Why can $RAWBIN_DIR be ignored when wrapping?  If the build system
> just sets $X during its wrappers, but the application needs $Y as well
> (wrapped in a build phase), then if the extra wrapping is cancelled,
> then the application won't get its $Y variable, which seems like a bug
> to me.
> 
> Greetings,
> Maxime.

It should be pretty easy to detect a wrapped binary without moving it.
We could just include a magic string in it and scan for that, the exact
same way we scan for store references. If it contains the "wrapped"
magic UUID, it's wrapped.

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Donnerstag, dem 28.04.2022 um 21:57 +0200 schrieb Maxime Devos:
> Liliana Marie Prikler schreef op do 28-04-2022 om 19:27 [+0200]:
> > > How does this help with double wrapping?  Whether the wrappers /
> > > originals are put in /bin or $RAWBIN_DIR, it's still wrapped twice.
> > Because $RAWBIN_DIR can be ignored when wrapping.  This means that
> > stuff that's already in it won't be added again.
> > [...]
> > Constructing the wrapper is not so much the problem, it's not
> > wrapping the already wrapped binaries. 
> 
> Why can $RAWBIN_DIR be ignored when wrapping?  If the build system just
> sets $X during its wrappers, but the application needs $Y as well
> (wrapped in a build phase), then if the extra wrapping is cancelled,
> then the application won't get its $Y variable, which seems like a bug
> to me.
The extra wrapping isn't cancelled though?  You just append the
definition of $Y to the the already existing definitions, but you don't
move the wrapper to $RAWBIN_DIR, because the actual binary already
exists there.

In other words, you have after wrap:
- bin/foo ~> rawbin/foo
and after wrap-again
- bin/foo ~> rawbin/foo
where ~> is the wrapping relation.

Currently, you have after one wrap
- bin/foo ~> bin/.foo-real
after two
- bin/foo ~> bin/.foo-real
- bin/.foo-real ~> bin/..foo-real-real
after three
- bin/foo ~> bin/.foo-real
- bin/.foo-real ~> bin/..foo-real-real
- bin/..foo-real-real ~> bin/...foo-real-real-real
and so on.

Is this clearer now?

Cheers

Re: The case for moving raw binaries

[zimoun]

Hi,

On Thu, 28 Apr 2022 at 18:37, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

> "raw binaries" (henceforth rawbins) are the unwrapped binaries that
> Guix leaves behind in $PACKAGE/bin with the .$WRAPPER-real name.  This
> practise causes several issues.  For one, those rawbins are visible in
> the shell by typing a dot and using tab completion.  What's more, in
> some build systems there might be two (or even more) off them.  This
> makes a generic wrap after wrap pattern almost impossible to achieve.

Could you provide more details or pointers about «several issues»?

For instance, I do not consider that these “rawbins” visible from shell
is an issue.  Why do you consider it is one?

And I do not understand what you mean by « makes a generic wrap after
wrap pattern».  Could you explain?


> So, what's the fix?  I propose moving rawbins to a different location.
> libexec would spring to mind as a place in which we could hide them, so
> would a new directory in the root of $PACKAGE.  Other than that, adding
> a rawbin output would also be possible, but I am not certain whether
> that'd be the right tradeoff.

Since I do not understand well the problems, contrary, I find “easier”
to have these “rawbins“ in the output store item.  But maybe, I am miss
a key point.


Cheers,
simon

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

Liliana Marie Prikler schreef op vr 29-04-2022 om 06:13 [+0200]:
> The extra wrapping isn't cancelled though?  You just append the
> definition of $Y to the the already existing definitions, but you don't
> move the wrapper to $RAWBIN_DIR, because the actual binary already
> exists there.

As I understood it, the suggestion was to cancel wrapping if there's
already things in $RAWBIN_DIR.

> In other words, you have after wrap:
> - bin/foo ~> rawbin/foo
> and after wrap-again
> - bin/foo ~> rawbin/foo
> where ~> is the wrapping relation.
> 
> Currently, you have after one wrap
> - bin/foo ~> bin/.foo-real
> after two
> - bin/foo ~> bin/.foo-real
> - bin/.foo-real ~> bin/..foo-real-real
> after three
> - bin/foo ~> bin/.foo-real
> - bin/.foo-real ~> bin/..foo-real-real
> - bin/..foo-real-real ~> bin/...foo-real-real-real
> and so on.
> 
> Is this clearer now?

I thought that

  (if already-wrapped?
      ;; PROG is already a wrapper: add the new "export VAR=VALUE"
      ;; lines just before the last line.
      [...])

in 'wrap-program' would avoid creating ..foo-real-real?

Also, doesn't 'wrap-program' refuse to wrap .foo-real:

  (when (wrapped-program? prog)
    (error (string-append prog " is a wrapper. Refusing to wrap.")))

so ..foo-real-real shouldn't be created in the first place?
Do you have a concrete example in which ..foo-real-real happens?

That said, the proposed new behaviour seems reasonable to me -- "pidof
emacs" would then actually find Emacs.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

raingloom schreef op vr 29-04-2022 om 00:52 [+0200]:
> It should be pretty easy to detect a wrapped binary without moving it.
> We could just include a magic string in it and scan for that, the exact
> same way we scan for store references. If it contains the "wrapped"
> magic UUID, it's wrapped.

This is already the case, at least for 'wrap-program':

(define (wrapped-program? prog)
  "Return #t if PROG is a program that was moved and wrapped by 'wrap-
program'."
  [...])

[...]
  (when (wrapped-program? prog)
    (error (string-append prog " is a wrapper. Refusing to wrap.")))


  (if already-wrapped?

      ;; PROG is already a wrapper: add the new "export VAR=VALUE"
lines just
      ;; before the last line.
      [...])
[...]

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Freitag, dem 29.04.2022 um 11:27 +0200 schrieb Maxime Devos:
> [...]
> 
> I thought that
> 
>   (if already-wrapped?
>       ;; PROG is already a wrapper: add the new "export VAR=VALUE"
>       ;; lines just before the last line.
>       [...])
> 
> in 'wrap-program' would avoid creating ..foo-real-real?
You are correct, I was going on old info that I haven't checked since.

This leaves us with
> That said, the proposed new behaviour seems reasonable to me --
> "pidof emacs" would then actually find Emacs.
and the annoyance that "." shell-completes to all the wrapped binaries.
For the former, there is IIRC still a bug in tramp (and I'm sure other
emacs packages), because a process name doesn't match the expected
regexp.

As for where to move things, I'm starting to lean a little closer
towards having an own output.  That way, we don't need to worry about
stuff from different directories (e.g. bin and sbin) shadowing each
other (even though that shouldn't occur), but more importantly, if we
need to copy data into rawbin so that it's correctly resolved, we can
do that.  The only thing that doesn't quite work is relative resolution
of commands, which would go through the wrapper-less binaries instead.
However, given that the wrapperless binary is invoked from a wrapped
binary, I am 73.69% certain, that this ought not to create too much of
a problem w.r.t. the set environment variables.

WDYT?

Re: The case for moving raw binaries

[Ludovic Courtès]

Hi!

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> "raw binaries" (henceforth rawbins) are the unwrapped binaries that
> Guix leaves behind in $PACKAGE/bin with the .$WRAPPER-real name.  This
> practise causes several issues.  For one, those rawbins are visible in
> the shell by typing a dot and using tab completion.  What's more, in
> some build systems there might be two (or even more) off them.  This
> makes a generic wrap after wrap pattern almost impossible to achieve.

One solution we haven’t yet used to its full potential is ‘wrap-script’,
an alternative to ‘wrap-program’ that Ricardo added to (guix build
utils) a while back.

Its advantage is that it does not create a new file.  It only works for
scripts, and only in some languages, but using it more widely may
already improve the situation noticeably.

Ludo’.

Re: The case for moving raw binaries

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

Ludovic Courtès schreef op ma 23-05-2022 om 17:10 [+0200]:
> One solution we haven’t yet used to its full potential is ‘wrap-script’,
> an alternative to ‘wrap-program’ that Ricardo added to (guix build
> utils) a while back.
> 
> Its advantage is that it does not create a new file.  It only works for
> scripts, and only in some languages, but using it more widely may
> already improve the situation noticeably.

Some time ago I wrote a variant of 'wrap-script' for perl scripts -- it
supported adjusting the load paths without touching any environment
variables, so it doesn't interfere with subprocesses (in contrast to
'wrap-program'), but setting arbitrary environment variables is not
(yet) supported.  It wasn't suitable for general usage yet and was
unpublished, but if anyone is interested, I could dig through git
branches & stashes to find it again.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Montag, dem 23.05.2022 um 17:10 +0200 schrieb Ludovic Courtès:
> Hi!
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
> 
> > "raw binaries" (henceforth rawbins) are the unwrapped binaries that
> > Guix leaves behind in $PACKAGE/bin with the .$WRAPPER-real name. 
> > This practise causes several issues.  For one, those rawbins are
> > visible in the shell by typing a dot and using tab completion. 
> > What's more, in some build systems there might be two (or even
> > more) off them. 
> > This makes a generic wrap after wrap pattern almost impossible to
> > achieve.
> 
> One solution we haven’t yet used to its full potential is ‘wrap-
> script’, an alternative to ‘wrap-program’ that Ricardo added to (guix
> build utils) a while back.
> 
> Its advantage is that it does not create a new file.  It only works
> for scripts, and only in some languages, but using it more widely may
> already improve the situation noticeably.
Speaking of wrap-script, for Guile in particular wouldn't it make sense
to use sh as a wrap-script wrapper with the #! exec hack?

Re: The case for moving raw binaries

[Philip McGrath]

Hi,

On Fri, Apr 29, 2022, at 12:59 PM, Liliana Marie Prikler wrote:
> Am Freitag, dem 29.04.2022 um 11:27 +0200 schrieb Maxime Devos:
>> [...]
>> 
>> I thought that
>> 
>>   (if already-wrapped?
>>       ;; PROG is already a wrapper: add the new "export VAR=VALUE"
>>       ;; lines just before the last line.
>>       [...])
>> 
>> in 'wrap-program' would avoid creating ..foo-real-real?
> You are correct, I was going on old info that I haven't checked since.
>
> This leaves us with
>> That said, the proposed new behaviour seems reasonable to me --
>> "pidof emacs" would then actually find Emacs.
> and the annoyance that "." shell-completes to all the wrapped binaries.
> For the former, there is IIRC still a bug in tramp (and I'm sure other
> emacs packages), because a process name doesn't match the expected
> regexp.
>
> As for where to move things, I'm starting to lean a little closer
> towards having an own output.  That way, we don't need to worry about
> stuff from different directories (e.g. bin and sbin) shadowing each
> other (even though that shouldn't occur), but more importantly, if we
> need to copy data into rawbin so that it's correctly resolved, we can
> do that.  The only thing that doesn't quite work is relative resolution
> of commands, which would go through the wrapper-less binaries instead.
> However, given that the wrapperless binary is invoked from a wrapped
> binary, I am 73.69% certain, that this ought not to create too much of
> a problem w.r.t. the set environment variables.
>
> WDYT?

I was mildly annoyed recently with several programs that use the ".foo-real" name in their `--help` output, for example:

```
$ guix shell --pure reuse -- reuse -h
usage: .reuse-real [-h] [--debug] [--include-submodules]
```

I wondered about just changing `wrap-program` to put the real program at `.real/foo` instead of `.foo-real`. One advantage is that it wouldn't need any special cooperation like setting up an output or an environment variable.

-Philip

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Freitag, dem 29.07.2022 um 17:20 -0400 schrieb Philip McGrath:
> Hi,
> 
> [...]
> I was mildly annoyed recently with several programs that use the
> ".foo-real" name in their `--help` output, for example:
> 
> ```
> $ guix shell --pure reuse -- reuse -h
> usage: .reuse-real [-h] [--debug] [--include-submodules]
> ```
> 
> I wondered about just changing `wrap-program` to put the real program
> at `.real/foo` instead of `.foo-real`. One advantage is that it
> wouldn't need any special cooperation like setting up an output or an
> environment variable.
Even as the one who made the suggestion that issue has an easier
workaround: Use exec -a to pass the 0th argument unchanged.

Cheers

Re: The case for moving raw binaries

[Ludovic Courtès]

Hi,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Freitag, dem 29.07.2022 um 17:20 -0400 schrieb Philip McGrath:
>> Hi,
>> 
>> [...]
>> I was mildly annoyed recently with several programs that use the
>> ".foo-real" name in their `--help` output, for example:
>> 
>> ```
>> $ guix shell --pure reuse -- reuse -h
>> usage: .reuse-real [-h] [--debug] [--include-submodules]
>> ```
>> 
>> I wondered about just changing `wrap-program` to put the real program
>> at `.real/foo` instead of `.foo-real`. One advantage is that it
>> wouldn't need any special cooperation like setting up an output or an
>> environment variable.
> Even as the one who made the suggestion that issue has an easier
> workaround: Use exec -a to pass the 0th argument unchanged.

That’s already happening:

--8<---------------cut here---------------start------------->8---
$ head -3 < $(guix build reuse)/bin/reuse | tail -1
exec -a "$0" "/gnu/store/iwsddc43xqxz4ibncrd7cgv4qjdy0jjd-reuse-1.0.0/bin/.reuse-real" "$@"
--8<---------------cut here---------------end--------------->8---

I’m not sure why it doesn’t have the desired effect though.

Ludo’.

Re: The case for moving raw binaries

[Liliana Marie Prikler]

Am Donnerstag, dem 04.08.2022 um 10:54 +0200 schrieb Ludovic Courtès:
> Hi,
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:
> 
> > Am Freitag, dem 29.07.2022 um 17:20 -0400 schrieb Philip McGrath:
> > > Hi,
> > > 
> > > [...]
> > > I was mildly annoyed recently with several programs that use the
> > > ".foo-real" name in their `--help` output, for example:
> > > 
> > > ```
> > > $ guix shell --pure reuse -- reuse -h
> > > usage: .reuse-real [-h] [--debug] [--include-submodules]
> > > ```
> > > 
> > > I wondered about just changing `wrap-program` to put the real
> > > program
> > > at `.real/foo` instead of `.foo-real`. One advantage is that it
> > > wouldn't need any special cooperation like setting up an output
> > > or an
> > > environment variable.
> > Even as the one who made the suggestion that issue has an easier
> > workaround: Use exec -a to pass the 0th argument unchanged.
> 
> That’s already happening:
> 
> --8<---------------cut here---------------start------------->8---
> $ head -3 < $(guix build reuse)/bin/reuse | tail -1
> exec -a "$0" "/gnu/store/iwsddc43xqxz4ibncrd7cgv4qjdy0jjd-reuse-
> 1.0.0/bin/.reuse-real" "$@"
> --8<---------------cut here---------------end--------------->8---
> 
> I’m not sure why it doesn’t have the desired effect though.
I found out: it's because reuse is a python script and the exec -a hack
stops working because python initializes sys.args using the filename.

Cheers