Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Hi,

There's a problem with the following commit:

> commit 7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99
> Author: John Doe <dftxbs3e@free.fr>
> Date:   Tue Dec 15 10:23:44 2020 +0100
> 
>   gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.
>   
>   Fixes <https://bugs.gnu.org/45252>.
>   
>   * gnu/packages/patches/libffi-float128-powerpc64le.patch: Import patch file
>   from <https://github.com/libffi/libffi/pull/561.patch>.
>   * gnu/packages/libffi.scm (libffi)[arguments]: Apply patch conditionally for
>   powerpc64le-* systems in a phase.
>   [inputs]: Add patch as input conditionally for powerpc64le-* systems.
>   * gnu/local.mk (dist_patch_DATA): Add patch file to build system.
>   
>   Signed-off-by: Chris Marusich <cmmarusich@gmail.com>

The problem is in how the 'patch' program is invoked, here:

> diff --git a/gnu/packages/libffi.scm b/gnu/packages/libffi.scm
> index d324892330..66239e0363 100644
> --- a/gnu/packages/libffi.scm
> +++ b/gnu/packages/libffi.scm
[...]
> @@ -67,13 +68,28 @@
>                                                        "powerpc-patch")))
>                                  (invoke "patch" "--batch" "-p1"
>                                          "-i" patch))))))
> +             '())
> +       ,@(if (string-prefix? "powerpc64le-" (or (%current-target-system)
> +                                                (%current-system)))
> +             '(#:phases (modify-phases %standard-phases
> +                          (add-after 'unpack 'apply-patch2
> +                            (lambda* (#:key inputs #:allow-other-keys)
> +                              (let ((patch (assoc-ref inputs
> +                                                      "powerpc64le-patch")))
> +                                (invoke "patch" "--batch" "-p1"
> +                                        "-i" patch))))))
>               '())))

When invoking 'patch' in Guix, you should *always* use "--force" instead
of "--batch".  There's a crucial difference between these two options:
If 'patch' finds that the given patch has already been applied, then
"--batch" will automatically *revert* the patch, whereas "--force" will
raise an error.  Here's the relevant section of the 'diffutils' manual:

> 10.11.2 Inhibiting Keyboard Input
> ---------------------------------
> 
> There are two ways you can prevent 'patch' from asking you any
> questions.  The '--force' ('-f') option assumes that you know what you
> are doing.  It causes 'patch' to do the following:
> 
>    * Skip patches that do not contain file names in their headers.
> 
>    * Patch files even though they have the wrong version for the
>      'Prereq:' line in the patch;
> 
>    * Assume that patches are not reversed even if they look like they
>      are.
> 
> The '--batch' ('-t') option is similar to '-f', in that it suppresses
> questions, but it makes somewhat different assumptions:
> 
>    * Skip patches that do not contain file names in their headers (the
>      same as '-f').
> 
>    * Skip patches for which the file has the wrong version for the
>      'Prereq:' line in the patch;
> 
>    * Assume that patches are reversed if they look like they are.

Now consider what will happen when we upgrade 'libffi' to a newer
version that already includes this fix.  If the Guix developer who
performs the upgrade forgets to remove this patch, the 'patch'
invocation above will start silently re-inserting the old bug.

We ran into this exact problem in the early years of Guix, and
henceforth changed all of the invocations of 'patch' to use '--force'.

Can we fix this right away, before many powerpc64le-* binaries are built
on top of it?

In any case, thanks very much for working on the powerpc64le port!

      Regards,
        Mark

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Earlier, I wrote:
> When invoking 'patch' in Guix, you should *always* use "--force" instead
> of "--batch".

(See <https://bugs.gnu.org/45252#19> for my earlier message).

Since writing the message above, I've found another problem in the same
commit (7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99): it searches for the
'patch' program in 'inputs'.  This is a mistake, because when
cross-compiling, 'inputs' will contain software compiled to run on the
target system instead of the build system.

If 'native-inputs' is not #f, we should search for the 'patch' program
in 'native-inputs' instead of 'inputs'.  Unless there's now a better way
that I don't know, I suggest adding 'native-inputs' to the list of
keyword arguments accepted by the 'lambda*', and changing 'inputs' to
"(or native-inputs inputs)" in the call to 'assoc-ref'.  Something like
this (untested):

_ ,@(if (string-prefix? "powerpc64le-" (or (%current-target-system)
__________________________________________ (%current-system)))
_______ '(#:phases (modify-phases %standard-phases
____________________ (add-after 'unpack 'apply-patch2
______________________ (lambda* (#:key inputs native-inputs
________________________________ #:allow-other-keys)
________________________ (let ((patch (assoc-ref (or native-inputs inputs)
________________________________________________ "powerpc64le-patch")))
__________________________ (invoke "patch" "--force" "-p1"
__________________________________ "-i" patch))))))
_______ '())

I see now that both of these mistakes were already present in the
"powerpc-*" case immediately above the recently-added "powerpc64le-*"
case.  The "powerpc-*" case was added earlier this year in the following
commit:

  https://git.sv.gnu.org/cgit/guix.git/commit/?id=02f5ee01c96589fc13f1e21b85b0b48100aec4e8

I'm CC'ing 'guix-devel' to raise awareness about these common mistakes,
in the hopes that reviewers will be more likely to notice them in the
future.

      Thanks,
        Mark

[bug#45252] [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5220 bytes --]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> Earlier, I wrote:
>> When invoking 'patch' in Guix, you should *always* use "--force" instead
>> of "--batch".
>
> (See <https://bugs.gnu.org/45252#19> for my earlier message).

Thank you for letting me know about this.  I didn't know about the
difference between "--batch" and "--force".  I agree we should use
"--force" instead of "--batch".  How do you recommend that I proceed?

I can definitely make another commit on the master branch to change the
option from "--batch" to "--force".  However, I'm reluctant to change
the option in the existing code on the master branch (introduced in
commit 02f5ee01c96589fc13f1e21b85b0b48100aec4e8), since I'm not sure how
many packages would be rebuilt as a result.  The powerpc64le
architecture is not bootstrapped at all yet, so it's not a problem for
that architecture, but I don't know the status for all the other
architectures beginning with "powerpc".

Before I make a new commit to change the patch option on the master
branch, I'd appreciate your advice on how to proceed.  Do you think it
would be better to make a commit on the master branch to fix just the
option I introduced in commit 7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99?
Or, do you think it would be better to make a commit on the core-updates
branch to change all the "--batch" options to "--force" options in the
libffi package definition?  If we did it on core-updates, we could just
replace the manual invocation of the "patch" tool with a change that
looks more like 4fff5ab24126a152b50c036b9bf8dc6f2740f094, in particular
this part:

diff --git a/gnu/packages/libffi.scm b/gnu/packages/libffi.scm
index 0e6a31d78c..0db8fa3e82 100644
--- a/gnu/packages/libffi.scm
+++ b/gnu/packages/libffi.scm
@@ -51,7 +51,8 @@
               (sha256
                (base32
                 "0mi0cpf8aa40ljjmzxb7im6dbj45bb0kllcd09xgmp834y9agyvj"))
-              (patches (search-patches "libffi-3.3-powerpc-fixes.patch"))))
+              (patches (search-patches "libffi-3.3-powerpc-fixes.patch"
+                                       "libffi-float128-powerpc64le.patch"))))
     (build-system gnu-build-system)
     (arguments
      `(;; Prevent the build system from passing -march and -mtune to the

I thought we wanted to apply patch 45252 on the master branch.  That's
why I made commit 7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99.  Afterwords,
I actually reverted commit 4fff5ab24126a152b50c036b9bf8dc6f2740f094 on
the core-updates branch in commit
b50341dba9811c048bed852c0279b828c7ddba66.  I reverted it because I
thought it would be undesirable to solve the same problem in two
different ways on two separate branches, and I thought that reverting it
would reduce the risk of merge conflicts later.

However, now that I think about it, I'm not sure the reversion was
necessary.  I'm actually not sure what the normal procedure is for
merging to/from core-updates and master (I've done many merges in my own
projects, but I've never done a merge in the Guix project), so I'm not
sure how a "TODO" task like the one mentioned in commit
7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99 ("Inline patches on next
rebuild cycle") would normally be resolved.  I would welcome any advice
you have about that.

By the way, a wip-ppc64le branch also exists, but I don't know what its
status is or whether I'm allowed to touch it.  I just assumed things
would be simpler if we applied patches to master branch when possible.

> Since writing the message above, I've found another problem in the same
> commit (7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99): it searches for the
> 'patch' program in 'inputs'.  This is a mistake, because when
> cross-compiling, 'inputs' will contain software compiled to run on the
> target system instead of the build system.

Is it searching for the "patch" program, or is it searching for the
patch file?  It looks to me like the code is searching for the patch
file in inputs, not the "patch" program.  The relevant code is here:

       ,@(if (string-prefix? "powerpc64le-" (or (%current-target-system)
                                                (%current-system)))
             '(#:phases (modify-phases %standard-phases
                          (add-after 'unpack 'apply-patch2
                            (lambda* (#:key inputs #:allow-other-keys)
                              (let ((patch (assoc-ref inputs
                                                      "powerpc64le-patch")))
                                (invoke "patch" "--batch" "-p1"
                                        "-i" patch))))))

The code invokes the "patch" program in the usual way.  My understanding
is that whatever version of the "patch" program that Guix has placed in
the PATH environment variable will be used.  Therefore, Guix will use
the correct "patch" program, regardless of whether or not the package is
being cross-compiled.  Am I misunderstanding something?

Again, thank you for taking the time to bring these topics up.  I'm
always trying to make sure I do things the best way I can in Guix, so I
appreciate the feedback.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 387 bytes --]

"regular powerpc", ie macppc/ppc32/powerpc-linux-gnu, does have some
bootstrap binaries built but isn't near ready for merging. Go ahead and
make any changes necessary.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:
> "regular powerpc", ie macppc/ppc32/powerpc-linux-gnu, does have some
> bootstrap binaries built but isn't near ready for merging. Go ahead and
> make any changes necessary.

I appreciate that, but if rebuilding the world on regular powerpc would
significantly add to the burden of even a single developer, then it's
probably not worth it.  I suggested fixing the powerpc64le case now only
because it was just added a few days ago, and more generally to raise
awareness about how best to run the 'patch' program in Guix.

If it's truly no extra burden, then you could change "--batch" to
"--force" on line 69 of libffi.c (in the "powerpc-*" case).

      Regards,
        Mark

Re: [bug#45252] [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le., [bug#45252] [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le., [bug#45252] [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2803 bytes --]

Hi,

Efraim Flashner <efraim@flashner.co.il> writes:

> "regular powerpc", ie macppc/ppc32/powerpc-linux-gnu, does have some
> bootstrap binaries built but isn't near ready for merging. Go ahead and
> make any changes necessary.

Mark H Weaver <mhw@netris.org> writes:

> Hi Efraim,
>
> Efraim Flashner <efraim@flashner.co.il> writes:
>> "regular powerpc", ie macppc/ppc32/powerpc-linux-gnu, does have some
>> bootstrap binaries built but isn't near ready for merging. Go ahead and
>> make any changes necessary.
>
> I appreciate that, but if rebuilding the world on regular powerpc would
> significantly add to the burden of even a single developer, then it's
> probably not worth it.  I suggested fixing the powerpc64le case now only
> because it was just added a few days ago, and more generally to raise
> awareness about how best to run the 'patch' program in Guix.
>
> If it's truly no extra burden, then you could change "--batch" to
> "--force" on line 69 of libffi.c (in the "powerpc-*" case).

OK.  I've made this change on master in commit
662e7e28d576ada91fc9dec7d27c100666114f03.

Mark H Weaver <mhw@netris.org> writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Earlier, I wrote:
>>>> When invoking 'patch' in Guix, you should *always* use "--force" instead
>>>> of "--batch".
>>>
>>> (See <https://bugs.gnu.org/45252#19> for my earlier message).
>>
>> Thank you for letting me know about this.  I didn't know about the
>> difference between "--batch" and "--force".  I agree we should use
>> "--force" instead of "--batch".  How do you recommend that I proceed?
>
> Simply changing "--batch" to "--force" on line 79 (in the powerpc64le-*
> case, i.e. the one that was just added) seems like the right thing.
> That will force a rebuild of almost everything on the powerpc64le-*
> architecture, but should not cause any rebuilds on other architectures.

OK, I've made this change on master in commit
fdb90e9ee8a578c88ef3a33067e8a532e43ae7b8.

>>> Since writing the message above, I've found another problem in the same
>>> commit (7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99): it searches for the
>>> 'patch' program in 'inputs'.  This is a mistake, because when
>>> cross-compiling, 'inputs' will contain software compiled to run on the
>>> target system instead of the build system.
>>
>> Is it searching for the "patch" program, or is it searching for the
>> patch file?  It looks to me like the code is searching for the patch
>> file in inputs, not the "patch" program.
>
> LOL, you're right, I got confused.  Please disregard my second email in
> this thread, and apologies for that noise.

No worries!  Thanks again for your help.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Earlier, I wrote:
>>> When invoking 'patch' in Guix, you should *always* use "--force" instead
>>> of "--batch".
>>
>> (See <https://bugs.gnu.org/45252#19> for my earlier message).
>
> Thank you for letting me know about this.  I didn't know about the
> difference between "--batch" and "--force".  I agree we should use
> "--force" instead of "--batch".  How do you recommend that I proceed?

Simply changing "--batch" to "--force" on line 79 (in the powerpc64le-*
case, i.e. the one that was just added) seems like the right thing.
That will force a rebuild of almost everything on the powerpc64le-*
architecture, but should not cause any rebuilds on other architectures.

>> Since writing the message above, I've found another problem in the same
>> commit (7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99): it searches for the
>> 'patch' program in 'inputs'.  This is a mistake, because when
>> cross-compiling, 'inputs' will contain software compiled to run on the
>> target system instead of the build system.
>
> Is it searching for the "patch" program, or is it searching for the
> patch file?  It looks to me like the code is searching for the patch
> file in inputs, not the "patch" program.

LOL, you're right, I got confused.  Please disregard my second email in
this thread, and apologies for that noise.

> Again, thank you for taking the time to bring these topics up.  I'm
> always trying to make sure I do things the best way I can in Guix, so I
> appreciate the feedback.

Thank you, Chris.

    Warm regards,
        Mark