[bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

[bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

[Efraim Flashner]

* guix/build/python-build-system.scm (ensure-no-cythonized-files): New procedure.
(%standard-phases): Add it.
---

This patch only checks if a file exists and suggests it may be
cythonized. It doesn't check the first line for the string "generated by
Cython" in that file or in any file. It also doesn't error when it
thinks it finds a file. It does provide a good visual queue that the
source needs more scrutiny before being added.

---
 guix/build/python-build-system.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 4a07496d22..bc34e22df8 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2019, 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
+;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -262,6 +263,18 @@ installed with setuptools."
   (setenv "PYTHONHASHSEED" "0")
   #t)
 
+(define* (ensure-no-cythonized-files #:rest _)
+  "Check the source code for @code{.c} files which may have been pre-generated
+by Cython."
+  (for-each
+    (lambda (file)
+      (let ((generated-file
+              (string-append (string-drop-right file 3) "c")))
+        (when (file-exists? generated-file)
+          (warning (G_ "Possible Cythonized file found: ~a~%") generated-file))))
+    (find-files "." "\\.pyx$"))
+  #t)
+
 (define %standard-phases
   ;; The build phase only builds C extensions and copies the Python sources,
   ;; while the install phase copies then byte-compiles the sources to the
@@ -271,6 +284,8 @@ installed with setuptools."
     (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980)
     (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism
       enable-bytecode-determinism)
+    (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files
+      ensure-no-cythonized-files)
     (delete 'bootstrap)
     (delete 'configure)                 ;not needed
     (replace 'build build)
-- 
2.29.2

bug#44590: build/python: Check for cythonized files

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Patch pushed

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

[Simon South]

Efraim,

I'm seeing an issue I think is related to this change. Trying to build
glib in core-updates today fails for me with "Unbound variable: warning"
in python-build-system.scm:

    starting phase `ensure-no-cythonized-files'
    Backtrace:
               9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
    In ice-9/eval.scm:
       191:35  8 (_ _)
    In guix/build/gnu-build-system.scm:
        886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
    In ice-9/boot-9.scm:
      1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
    In srfi/srfi-1.scm:
        634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
    In guix/build/gnu-build-system.scm:
       895:33  4 (_ _)
    In guix/build/python-build-system.scm:
        272:2  3 (ensure-no-cythonized-files . _)
    In srfi/srfi-1.scm:
        634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
    In guix/build/python-build-system.scm:
       277:10  1 (_ _)
    In ice-9/boot-9.scm:
      1669:16  0 (raise-exception _ #:continuable? _)

    ice-9/boot-9.scm:1669:16: In procedure raise-exception:
    Unbound variable: warning
    builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
    build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
    View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
    cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
    guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed

And in fact building python-build-system.scm shows

    [100%] GUILEC   guix/build/python-build-system.go
    WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
    guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
    guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'

Adding use-module directives for (guix diagnostics) and (guix i18n) to
the module doesn't seem to fix the problem, presumably (if I understand
correctly) because these modules are not then also imported on the build
side of things.

Are you aware of this issue?

-- 
Simon South
simon@simonsouth.net

[bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2875 bytes --]

On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I tested it mostly against master. I'll see if I can figure out how I
broke it.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#44590] [WIP PATCH CORE-UPDATES] build/python: Check for cythonized files.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]

On Sun, Dec 13, 2020 at 11:53:31AM -0500, Simon South wrote:
> Efraim,
> 
> I'm seeing an issue I think is related to this change. Trying to build
> glib in core-updates today fails for me with "Unbound variable: warning"
> in python-build-system.scm:
> 
>     starting phase `ensure-no-cythonized-files'
>     Backtrace:
>                9 (primitive-load "/gnu/store/ycv4j8dqk44g6d72m2ykvlwb9wc?")
>     In ice-9/eval.scm:
>        191:35  8 (_ _)
>     In guix/build/gnu-build-system.scm:
>         886:2  7 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
>     In ice-9/boot-9.scm:
>       1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>     In srfi/srfi-1.scm:
>         634:9  5 (for-each #<procedure 7ffff4f662e0 at guix/build/gnu-b?> ?)
>     In guix/build/gnu-build-system.scm:
>        895:33  4 (_ _)
>     In guix/build/python-build-system.scm:
>         272:2  3 (ensure-no-cythonized-files . _)
>     In srfi/srfi-1.scm:
>         634:9  2 (for-each #<procedure 7ffff5ad3978 at guix/build/pytho?> ?)
>     In guix/build/python-build-system.scm:
>        277:10  1 (_ _)
>     In ice-9/boot-9.scm:
>       1669:16  0 (raise-exception _ #:continuable? _)
> 
>     ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>     Unbound variable: warning
>     builder for `/gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv' failed with exit code 1
>     build of /gnu/store/cgmgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv failed
>     View build log at '/var/log/guix/drvs/cg/mgrddwjvryxmj4nn3naiwy42yi3wzn-meson-for-build-0.55.1.drv.bz2'.
>     cannot build derivation `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv': 1 dependencies couldn't be built
>     guix build: error: build of `/gnu/store/a00wr5wj4cd0b98ixxynyxgbw2nyv8kg-glib-2.62.6.drv' failed
> 
> And in fact building python-build-system.scm shows
> 
>     [100%] GUILEC   guix/build/python-build-system.go
>     WARNING: (guix build python-build-system): imported module (guix build utils) overrides core binding `delete'
>     guix/build/python-build-system.scm:277:10: warning: possibly unbound variable `warning'
>     guix/build/python-build-system.scm:277:19: warning: possibly unbound variable `G_'
> 
> Adding use-module directives for (guix diagnostics) and (guix i18n) to
> the module doesn't seem to fix the problem, presumably (if I understand
> correctly) because these modules are not then also imported on the build
> side of things.
> 
> Are you aware of this issue?
> 

I fixed it. I changed the code to output using format and (ice-9 format)
and I tested it building meson-for-build and python-setuptools.

I suppose I should point out now that meson-for-build has some possibly
cythonized files :)

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]