Python Tox Dependencies Invisible to Pip

Python Tox Dependencies Invisible to Pip

[Antwane Mason]

I am currently trying to build a package for python-virtualenv-clone with
the eventual goal of submitting a patch. While building the package, tox is
used to perform tests and it needs but cannot find pytest and virtualenv
despite being included as build dependencies. I suspect that tox has done
something to the environment variables guix uses to expose python packages
from store. However, I have no experience with tox and need some guidance
on how to fix this. Any help would be appreciated. I still haven't been
able to build onlykey-cli package due to chasing down dependencies such as
this package.

Here is a tox log showing that pip cannot find pytest dependency.
------------------------------------------------
action: py38, msg: getenv
cwd: /tmp/guix-build-python-virtualenv-clone-0.5.6.drv-0/source
cmd:
/tmp/guix-build-python-virtualenv-clone-0.5.6.drv-0/source/.tox/py38/bin/python
-m pip install pytest virtualenv
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent
directory is not owned or is not writable by the current user. The cache
has been disabled. Check the permissions and owner of that directory. If
executing pip with sudo, you may want sudo's -H flag.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff5faab50>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/pytest/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff5fbe730>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/pytest/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff5fbe6d0>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/pytest/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff5fbe8e0>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/pytest/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff5fbeaf0>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/pytest/
ERROR: Could not find a version that satisfies the requirement pytest (from
versions: none)
ERROR: No matching distribution found for pytest
-----------------------------------------------

Here is a snippet from build output that makes me think tox is discarding
the environment variable that points to python packages in guix store.

-----------------------------------------------
py38 installdeps: pytest, virtualenv
WARNING: Discarding $PYTHONPATH from environment, to override specify
PYTHONPATH in 'passenv' in your configuration.
-----------------------------------------------

Here is the package definition. I had to change the usual style of using
pypi to get code because the tarball in pypi is missing tox.ini that can be
found in the github tarball for the same version. This can be verified by
checking the contents and you can also see a difference in hashes between
github and pypi. I plan on opening an issue on their github repo soon on
this.
------------------------------------------------
(define-public python-virtualenv-clone
  (package
    (name "python-virtualenv-clone")
    (version "0.5.6")
    (source
      (origin
        (method git-fetch)
        (uri
         (git-reference (url "
https://github.com/edwardgeorge/virtualenv-clone")
                        (commit version)))
        (sha256
          (base32
            "0xb20fhl99dw5vnyb43sjpj9628nbdnwp5g7m8f2id7w8kpwzvfw"))))
    (native-inputs
     `(("python-pytest" ,python-pytest)
       ("python-tox" ,python-tox)
       ("python-virtualenv" ,python-virtualenv)))
    (build-system python-build-system)
    (home-page
      "https://github.com/edwardgeorge/virtualenv-clone")
    (synopsis "script to clone virtualenvs.")
    (description "script to clone virtualenvs.")
    (license license:expat)))
------------------------------------------------

Regards,
Antwane

Re: Python Tox Dependencies Invisible to Pip

[Antwane Mason]

It turns out my intuition was right and I needed to prevent tox from
unsetting PYTHONPATH environment variable. The solution was to add a build
phase prior to testing that passes the PYTHONPATH environment variable to
the virtual environments. Unrelated to the problem, I also needed to tell
tox to skip interpreters that are not present on my system and I needed to
adjust the test command to point to the "tests" folder where the unit tests
were stored. Below is the resulting package definition. I'm one step closer
to building python-onlykey package but now having to fight with test
failures while building python-verspec, another transitive dependency of
python-onlykey.

------------------------------------------------
(define-public python-virtualenv-clone
  (package
    (name "python-virtualenv-clone")
    (version "0.5.6")
    (source
      (origin
        (method git-fetch)
        (uri
         (git-reference (url "
https://github.com/edwardgeorge/virtualenv-clone")
                        (commit version)))
        (sha256
          (base32
            "0xb20fhl99dw5vnyb43sjpj9628nbdnwp5g7m8f2id7w8kpwzvfw"))))
    (native-inputs
     `(("python-pytest" ,python-pytest)
       ("python-tox" ,python-tox)
       ("python-virtualenv" ,python-virtualenv)))
    (build-system python-build-system)
    (arguments
     '(#:phases
       (modify-phases %standard-phases
         (add-before 'check 'configure-tox
                    (lambda _
                      (substitute* "tox.ini"
                        (("(\\[testenv\\])" section-header)
                          (string-append section-header
                                         "\npassenv = PYTHONPATH"))
                        (("(\\[tox\\])" section-header)
                          (string-append section-header
                                         "\nskip_missing_interpreters =
true"))
                        (("(commands = py.test -v) \\[\\]" _ cmd)
                         (string-append cmd " tests")))
                      #t)))))
    (home-page
     "https://github.com/edwardgeorge/virtualenv-clone")
    (synopsis "script to clone virtualenvs.")
    (description "script to clone virtualenvs.")
    (license license:expat)))
------------------------------------------------
Regards,
Antwane