Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

[Marius Bakke]

Hi Guix,

I tried switching to GCC 11 on the core-updates branch, but it fails
early when attempting to repack the GCC source code for GCC-BOOT0,
because some files in its test suite contains non-ASCII characters:

--8<---------------cut here---------------start------------->8---
[... unpacking ...]
patching file gcc/builtins.c
Hunk #1 succeeded at 4623 with fuzz 1 (offset 1341 lines).
Hunk #2 succeeded at 6097 with fuzz 2 (offset 2206 lines).
patching file gcc/gimple-fold.c
Hunk #1 succeeded at 665 (offset 9 lines).
Hunk #2 succeeded at 766 with fuzz 2 (offset 16 lines).
patching file libvtv/Makefile.in
Hunk #1 succeeded at 14 with fuzz 1 (offset -1 lines).
source is at 'gcc-11.3.0'
applying '/gnu/store/g0ba4l825z9i4l1jd5cqvl6m09xicdwa-gcc-9-strmov-store-file-names.patch'...
applying '/gnu/store/5705r4ajxl8lav1hz9xm19w75zdcz1n2-gcc-5.0-libvtv-runpath.patch'...
find-files: gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go: No such file or directory
Backtrace:
In srfi/srfi-1.scm:
 465: 19 [fold #<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 18 [#<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 450: 17 [loop "gcc" "gcc-11.3.0" ...]
In srfi/srfi-1.scm:
 465: 16 [fold #<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 15 [#<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 450: 14 [loop "testsuite" "gcc-11.3.0/gcc" ...]
In srfi/srfi-1.scm:
 465: 13 [fold #<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 12 [#<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 450: 11 [loop "go.test" "gcc-11.3.0/gcc/testsuite" ...]
In srfi/srfi-1.scm:
 465: 10 [fold #<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 9 [#<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 450: 8 [loop "test" "gcc-11.3.0/gcc/testsuite/go.test" ...]
In srfi/srfi-1.scm:
 465: 7 [fold #<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 6 [#<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 450: 5 [loop "fixedbugs" "gcc-11.3.0/gcc/testsuite/go.test/test" ...]
In srfi/srfi-1.scm:
 465: 4 [fold #<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
In ice-9/ftw.scm:
 452: 3 [#<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
 474: 2 [loop "issue27836.dir" ...]
In guix/build/utils.scm:
 540: 1 [#<procedure d188e0 at guix/build/utils.scm:536:28 (file stat errno result)> "gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go" ...]
In unknown file:
   ?: 0 [scm-error misc-error #f "~A" ("find-files failed") #f]

ERROR: In procedure scm-error:
ERROR: find-files failed
--8<---------------cut here---------------end--------------->8---

Deleting these files also don't work for the same reason, even when
using the hex representation, i.e. (delete-file "\u00c4foo.go"), or with
DELETE-FILE-RECURSIVELY.

One workaround is to avoid the use of BOOTSTRAP-ORIGIN by applying the
patches and snippet in phases, but that's suboptimal because it has to
be done for all of GCC-BOOT0, LIBSTDC++, and GCC-FINAL.

I'll try this workaround to get things going, but hoping for better
suggestions!

Re: Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

[Attila Lendvai]

you may find something in this issue that you can mimic, or it may inspire a higher level fix for both issues:

https://issues.guix.gnu.org/54893#12

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The state represents violence in a concentrated and organized form. The individual has a soul, but as the state is a soulless machine, it can never be weaned from violence to which it owes its very existence.”
	— Mahatma Gandhi (1869–1948)

Re: Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

[Greg Hogan]

Marius,

Thank you for your work upgrading the core packages!

On the off chance that the following is helpful, in order to switch
the build to GCC 11 or 12 I had to apply the patch (with the missing
endif) from
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100017#c12

You may have avoided or worked around this issue, but even though a
different fix from the ticket was patched into our GCC 11.3 and 12.1,
these would not bootstrap for me without that patch.

Greg

On Mon, Jul 18, 2022 at 3:49 PM Marius Bakke <marius@gnu.org> wrote:
>
> Hi Guix,
>
> I tried switching to GCC 11 on the core-updates branch, but it fails
> early when attempting to repack the GCC source code for GCC-BOOT0,
> because some files in its test suite contains non-ASCII characters:
>
> --8<---------------cut here---------------start------------->8---
> [... unpacking ...]
> patching file gcc/builtins.c
> Hunk #1 succeeded at 4623 with fuzz 1 (offset 1341 lines).
> Hunk #2 succeeded at 6097 with fuzz 2 (offset 2206 lines).
> patching file gcc/gimple-fold.c
> Hunk #1 succeeded at 665 (offset 9 lines).
> Hunk #2 succeeded at 766 with fuzz 2 (offset 16 lines).
> patching file libvtv/Makefile.in
> Hunk #1 succeeded at 14 with fuzz 1 (offset -1 lines).
> source is at 'gcc-11.3.0'
> applying '/gnu/store/g0ba4l825z9i4l1jd5cqvl6m09xicdwa-gcc-9-strmov-store-file-names.patch'...
> applying '/gnu/store/5705r4ajxl8lav1hz9xm19w75zdcz1n2-gcc-5.0-libvtv-runpath.patch'...
> find-files: gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go: No such file or directory
> Backtrace:
> In srfi/srfi-1.scm:
>  465: 19 [fold #<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 18 [#<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 17 [loop "gcc" "gcc-11.3.0" ...]
> In srfi/srfi-1.scm:
>  465: 16 [fold #<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 15 [#<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 14 [loop "testsuite" "gcc-11.3.0/gcc" ...]
> In srfi/srfi-1.scm:
>  465: 13 [fold #<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 12 [#<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 11 [loop "go.test" "gcc-11.3.0/gcc/testsuite" ...]
> In srfi/srfi-1.scm:
>  465: 10 [fold #<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 9 [#<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 8 [loop "test" "gcc-11.3.0/gcc/testsuite/go.test" ...]
> In srfi/srfi-1.scm:
>  465: 7 [fold #<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 6 [#<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 5 [loop "fixedbugs" "gcc-11.3.0/gcc/testsuite/go.test/test" ...]
> In srfi/srfi-1.scm:
>  465: 4 [fold #<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 3 [#<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  474: 2 [loop "issue27836.dir" ...]
> In guix/build/utils.scm:
>  540: 1 [#<procedure d188e0 at guix/build/utils.scm:536:28 (file stat errno result)> "gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go" ...]
> In unknown file:
>    ?: 0 [scm-error misc-error #f "~A" ("find-files failed") #f]
>
> ERROR: In procedure scm-error:
> ERROR: find-files failed
> --8<---------------cut here---------------end--------------->8---
>
> Deleting these files also don't work for the same reason, even when
> using the hex representation, i.e. (delete-file "\u00c4foo.go"), or with
> DELETE-FILE-RECURSIVELY.
>
> One workaround is to avoid the use of BOOTSTRAP-ORIGIN by applying the
> patches and snippet in phases, but that's suboptimal because it has to
> be done for all of GCC-BOOT0, LIBSTDC++, and GCC-FINAL.
>
> I'll try this workaround to get things going, but hoping for better
> suggestions!
>

Re: Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2922 bytes --]

Greg Hogan <code@greghogan.com> skriver:

> On the off chance that the following is helpful, in order to switch
> the build to GCC 11 or 12 I had to apply the patch (with the missing
> endif) from
>   https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100017#c12
>
> You may have avoided or worked around this issue, but even though a
> different fix from the ticket was patched into our GCC 11.3 and 12.1,
> these would not bootstrap for me without that patch.

Neat.  Do you have a patch submitted to the tracker already?  Did you
also find a solution to the file name problem?

I took a slightly different route to work around the libstdc++ bootstrap
issue (see bottom hunk):

--8<---------------cut here---------------start------------->8---
               (add-after 'unpack 'apply-patches-and-snippet
                 (lambda* (#:key inputs #:allow-other-keys)
                   (let ((patch1 (assoc-ref inputs "patch1"))
                         (patch2 (assoc-ref inputs "patch2"))
                         (libstdc++ (assoc-ref inputs "libstdc++")))

                     ;; Apply the patch inputs added below.
                     (for-each (lambda (patch)
                                 (invoke "patch" "-p1" "--input" patch
                                         "--no-backup-if-mismatch"))
                               (list patch1 patch2))

                     ;; Apply the gcc-canadian-cross-objdump snippet.
                     (substitute* "libcc1/configure"
                       (("\\$gcc_cv_objdump -T")
                        "$OBJDUMP_FOR_TARGET -T"))

                     ;; Fix a regression in GCC 11 where the libstc++ input
                     ;; shadows glibc headers when building libstdc++.  An
                     ;; upstream fix was added in GCC 11.3.0, but it only
                     ;; hides system include directories, not those on
                     ;; CPLUS_INCLUDE_PATH.  See discussion at
                     ;; <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100017>.
                     (substitute* "libstdc++-v3/src/c++17/Makefile.in"
                       (("^AM_CXXFLAGS = ")
                        (string-append "CPLUS_INCLUDE_PATH = "
                                       (string-join
                                        (remove (cut string-prefix? libstdc++ <>)
                                                (string-split
                                                 (getenv "CPLUS_INCLUDE_PATH")
                                                 #\:))
                                        ":")
                                       "\nAM_CXXFLAGS = "))))))))
--8<---------------cut here---------------end--------------->8---

It's not pretty, but does the job!  I won't push it right away though,
waiting for feedback and checking that things generally work first.

-- 
Thanks,
Marius

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Dealing with non-ASCII file names in BOOTSTRAP-ORIGIN

[Ludovic Courtès]

Hi!

Marius Bakke <marius@gnu.org> skribis:

> I tried switching to GCC 11 on the core-updates branch, but it fails
> early when attempting to repack the GCC source code for GCC-BOOT0,
> because some files in its test suite contains non-ASCII characters:
>
> [... unpacking ...]
> patching file gcc/builtins.c
> Hunk #1 succeeded at 4623 with fuzz 1 (offset 1341 lines).
> Hunk #2 succeeded at 6097 with fuzz 2 (offset 2206 lines).
> patching file gcc/gimple-fold.c
> Hunk #1 succeeded at 665 (offset 9 lines).
> Hunk #2 succeeded at 766 with fuzz 2 (offset 16 lines).
> patching file libvtv/Makefile.in
> Hunk #1 succeeded at 14 with fuzz 1 (offset -1 lines).
> source is at 'gcc-11.3.0'
> applying '/gnu/store/g0ba4l825z9i4l1jd5cqvl6m09xicdwa-gcc-9-strmov-store-file-names.patch'...
> applying '/gnu/store/5705r4ajxl8lav1hz9xm19w75zdcz1n2-gcc-5.0-libvtv-runpath.patch'...
> find-files: gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go: No such file or directory
> Backtrace:
> In srfi/srfi-1.scm:
>  465: 19 [fold #<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 18 [#<procedure 143cb10 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 17 [loop "gcc" "gcc-11.3.0" ...]
> In srfi/srfi-1.scm:
>  465: 16 [fold #<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 15 [#<procedure 2375210 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 14 [loop "testsuite" "gcc-11.3.0/gcc" ...]
> In srfi/srfi-1.scm:
>  465: 13 [fold #<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 12 [#<procedure ebad80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 11 [loop "go.test" "gcc-11.3.0/gcc/testsuite" ...]
> In srfi/srfi-1.scm:
>  465: 10 [fold #<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 9 [#<procedure 219a4b0 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 8 [loop "test" "gcc-11.3.0/gcc/testsuite/go.test" ...]
> In srfi/srfi-1.scm:
>  465: 7 [fold #<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 6 [#<procedure 229ba80 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  450: 5 [loop "fixedbugs" "gcc-11.3.0/gcc/testsuite/go.test/test" ...]
> In srfi/srfi-1.scm:
>  465: 4 [fold #<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> ...]
> In ice-9/ftw.scm:
>  452: 3 [#<procedure 20bab70 at ice-9/ftw.scm:451:38 (subdir result+visited)> # #]
>  474: 2 [loop "issue27836.dir" ...]
> In guix/build/utils.scm:
>  540: 1 [#<procedure d188e0 at guix/build/utils.scm:536:28 (file stat errno result)> "gcc-11.3.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go" ...]
> In unknown file:
>    ?: 0 [scm-error misc-error #f "~A" ("find-files failed") #f]
>
> ERROR: In procedure scm-error:
> ERROR: find-files failed
>
> Deleting these files also don't work for the same reason, even when
> using the hex representation, i.e. (delete-file "\u00c4foo.go"), or with
> DELETE-FILE-RECURSIVELY.
>
> One workaround is to avoid the use of BOOTSTRAP-ORIGIN by applying the
> patches and snippet in phases, but that's suboptimal because it has to
> be done for all of GCC-BOOT0, LIBSTDC++, and GCC-FINAL.

To be clear, you’re calling ‘delete-file-recursively’ from a snippet,
right?

If so, maybe you could… avoid it somehow?  :-)

Other options, as discussed on IRC, include adding ‘glibc-utf8-locales’
to the environment, or calling out to Coreutils as you proposed…

Ludo’.