Python's native-inputs

Python's native-inputs

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

Hi Guix,

On some languages, there are a lot of unused native-inputs that are
development & linting dependencies much more than packages that are
actually used to build or test a package (I'm thinking at least Python
and Rust). These fall in the category of tools "useful" at run time, but
unecessary at build time.

Is there a clear policy about their removal? I've seen the case of
pre-commit in Python, and I've commited a series yesterday regarding
pylint, but there are a whole lot of them in Python, at least :

- black and extensions
- codecov
- coveralls
- coverage
- pytest-cov
- pycodestyle
- mypy and extensions
- isort and extensions
- flake8 and extensions
- possibly virtualenv and extensions
and I'm probably forgetting some.

These packages make a lot of sense when considering things like
`guix shell -D` but they are hampering some progress on Python packages
since they are everywhere and a small update in their inputs rebuilds
the whole python world (even though it has NO influence on the
functionality of any other package).

What are the guidelines in this case?

I can propose a huge patch series (currently ~300 patches, and not
finished), to remove them, lint against them and remove them from the
importer as a default, but that's a big decision to make. IMO we should
have a dev-inputs field to handle these cases, but that's even more work.

-- 
Best regards,
Nicolas Graves

Re: Python's native-inputs

[Maxim Cournoyer]

Hi Nicolas,

Nicolas Graves via "Development of GNU Guix and the GNU System
distribution." <guix-devel@gnu.org> writes:

> Hi Guix,
>
> On some languages, there are a lot of unused native-inputs that are
> development & linting dependencies much more than packages that are
> actually used to build or test a package (I'm thinking at least Python
> and Rust). These fall in the category of tools "useful" at run time, but
> unecessary at build time.

Indeed.

> Is there a clear policy about their removal? I've seen the case of
> pre-commit in Python, and I've commited a series yesterday regarding
> pylint, but there are a whole lot of them in Python, at least :

[...]

> These packages make a lot of sense when considering things like
> `guix shell -D` but they are hampering some progress on Python packages
> since they are everywhere and a small update in their inputs rebuilds
> the whole python world (even though it has NO influence on the
> functionality of any other package).
>
> What are the guidelines in this case?

There aren't really any guidelines.  It's easy to avoid the linters, it
makes sense to avoid them, but sometimes upstream makes it difficult to
avoid running a linter as part of there test suite, in which case I
think it's acceptable to keep them as native-inputs rather than come up
with more patches to maintain.

> I can propose a huge patch series (currently ~300 patches, and not
> finished), to remove them, lint against them and remove them from the
> importer as a default, but that's a big decision to make. IMO we should
> have a dev-inputs field to handle these cases, but that's even more work.

The situation is similar as for other test inputs such as pytest; it has
an enormous amounts of dependents and thus cannot be easily upgraded on
the master branch.  At least more up-to-date variants can be added since
these are not in the transitive closure of any packages.

I don't think we should go out of our way to address annoyances that
upstream have caused -- that'd be too much work to maintain in the long
run.

-- 
Thanks,
Maxim

Re: Python's native-inputs

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

Answer in two different emails.

 On 2024-04-18 22:07, Maxim Cournoyer wrote:

> Hi Nicolas,
>
> Nicolas Graves via "Development of GNU Guix and the GNU System
> distribution." <guix-devel@gnu.org> writes:
>
>> Hi Guix,
>>
>> On some languages, there are a lot of unused native-inputs that are
>> development & linting dependencies much more than packages that are
>> actually used to build or test a package (I'm thinking at least Python
>> and Rust). These fall in the category of tools "useful" at run time, but
>> unecessary at build time.
>
> Indeed.

One more argument: it's important for user experience using python to
have these packages up-to-date, because users will want to have an
up-to-date lsp client and utilities.

>
>> Is there a clear policy about their removal? I've seen the case of
>> pre-commit in Python, and I've commited a series yesterday regarding
>> pylint, but there are a whole lot of them in Python, at least :
>
> [...]
>
>> These packages make a lot of sense when considering things like
>> `guix shell -D` but they are hampering some progress on Python packages
>> since they are everywhere and a small update in their inputs rebuilds
>> the whole python world (even though it has NO influence on the
>> functionality of any other package).
>>
>> What are the guidelines in this case?
>
> There aren't really any guidelines.  It's easy to avoid the linters, it
> makes sense to avoid them, but sometimes upstream makes it difficult to
> avoid running a linter as part of there test suite, in which case I
> think it's acceptable to keep them as native-inputs rather than come up
> with more patches to maintain.
>
>> I can propose a huge patch series (currently ~300 patches, and not
>> finished), to remove them, lint against them and remove them from the
>> importer as a default, but that's a big decision to make. IMO we should
>> have a dev-inputs field to handle these cases, but that's even more work.
>
> The situation is similar as for other test inputs such as pytest; it has
> an enormous amounts of dependents and thus cannot be easily upgraded on
> the master branch.  At least more up-to-date variants can be added since
> these are not in the transitive closure of any packages.

 That's true for some very essential packages (setuptools, pytest...)
 which would make the case to only update these packages once we want to
 update the whole python chain, i.e. when we update python. But IMO it
 shouldn't be the case for a package like black, isort or flake8.

>
> I don't think we should go out of our way to address annoyances that
> upstream have caused -- that'd be too much work to maintain in the long
> run.

 I think it depends on the case. For a package like python-dateutil,
 having pytest-cov as a native-input is bad, because it causes a rebuild
 of the whole chain (guix refresh -l python-pytest-cov went from ~2500
 to ~15 packages with the whole series, but in particular that was a big
 part of dependency removal). Now for small leaf packages, having
 pytest-cov as a native-input is not a big issue indeed, I agree in
 those cases we would better ignore it than fix it in the meantime.

 While making the patches, I realized it's not often that we need a
 convoluted patch. Most of the time, adding "-c" "/dev/null" to
 test-flags will do the trick to remove python-pytest-cov and it's even
 less annoyance for black, flake8 or mypy, if not other cases are often
 handled with a simple removal of the native-input in pyproject.toml, or
 setup.py. What's annoying is when packages are flagged as required when
 they're not.

 It could possibly even be handled in the build-system in the case of
 pytest, to avoid these "-c" "/dev/null". This is untested, but I think
 it'll work.
 
;; Add a pytest plugin for the pyproject-build-system to filter out some
;; arguments we want to be able to ignore.
;; (define (pytest-ignore-options group-options-alist)
;;   "This function converts an alist of (group . options) in a string that can
;; be instantiated as a python pytest plugin."
;;   (string-join
;;    (cons*
;;     "import pytest"
;;     (map
;;      (match-lambda
;;        ((group . options)
;;         (string-append
;;          (format #f "def pytest_addoption(parser):
;;     group = parser.getgroup(~s,'Guix ignored options')
;;     options = [~{~s, ~}]
;;     for option in options:
;;         group.addoption(option, action='count')"
;;                  group
;;                  options))))
;;      group-options-alist))
;;    "\n\n"))

;; (let ((cov-alist
;;        '(("cov" . ("--cov" "--cov-reset" "--cov-report" "--cov-config"
;;                    "--no-cov-on-fail" "--no-cov" "--cov-fail-under"
;;                    "--cov-append" "--cov-branch" "--cov-context")))))
;;   (display (pytest-ignore-options cov-alist)))

;; TODO Instantiate this file, add it to the check phase through the 
;; PYTEST_PLUGINS environment variable (possibly also PYTHONPATH or GUIX_PYTHONPATH ?)
;; reference: https://docs.pytest.org/en/7.1.x/how-to/writing_plugins.html#plugin-discovery-order-at-tool-startup
;; Using the environment variable should also set the right precedence :
;; if pytest-cov is in native-inputs, then use it; if not, ignore --cov* options.

 I'm not sure we can do the same for packages though, it seems that the
 proper is not easy to grasp.

 But in a near future, I wouldn't be surprised if we can tweak the
 pyproject-build-system to simply read the configuration and remove
 unwanted packages when they are present. It's definitely not available yet
 and my experiments didn't work, but python-pypa-build is headed in that
 direction for some of its utilities, and python-3.12 seems to also
 change the bootstrap order of its packages (python-pypa-build will be
 easier to bootstrap).

 TL;DR :
 - patch series in big progress, not done yet because I don't really
 know where to stop and massive rebuilds.
 - WDYT about tweaking the build-system for pytest?
 - not done : tweaking the pypi import to ignore those packages. I've
 done it in the juliahub import code, but I haven't delved into that
 though.
 - not done : adding those packages to guix lint - trivial. We should
 probably add a variable in pypi import that is exported and added into
 guix lint.

 Acting on the three (guix import, guix lint, guix build) can make it
 quite easy on maintainers in the end IMO.
 
-- 
Best regards,
Nicolas Graves

Re: Python's native-inputs

[Ricardo Wurmus]

>  TL;DR :
>  - patch series in big progress, not done yet because I don't really
>  know where to stop and massive rebuilds.

Please take a look at the python-team branch, which contains changes to
the build system.

-- 
Ricardo

Re: Python's native-inputs

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

On 2024-04-22 16:40, Ricardo Wurmus wrote:

>>  TL;DR :
>>  - patch series in big progress, not done yet because I don't really
>>  know where to stop and massive rebuilds.
>
> Please take a look at the python-team branch, which contains changes to
> the build system.

I'll rebase on it, thanks.

-- 
Best regards,
Nicolas Graves

Re: Python's native-inputs

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

Hi Guix,

Just to let you know, I just added a working patch series that does this
job on (guix build, guix import pypi, guix lint). This is not the full
patch series, which I have rebased on python-team, but it should be good
enough
1) to test it
2) to review it for a v2 that would be more guixy

It's in 70570.

Cheers,
Nicolas 

On 2024-04-22 16:40, Ricardo Wurmus wrote:

>>  TL;DR :
>>  - patch series in big progress, not done yet because I don't really
>>  know where to stop and massive rebuilds.
>
> Please take a look at the python-team branch, which contains changes to
> the build system.

-- 
Best regards,
Nicolas Graves

Re: Python's native-inputs

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

On 2024-04-25 18:03, Nicolas Graves wrote:

> Hi Guix,
>
> Just to let you know, I just added a working patch series that does this
> job on (guix build, guix import pypi, guix lint). This is not the full
> patch series, which I have rebased on python-team, but it should be good
> enough
> 1) to test it
> 2) to review it for a v2 that would be more guixy
>
> It's in 70570.
>
> Cheers,
> Nicolas

Since I couldn't send the whole patch series at once in 70735 and since
I had some rebasing to do due to Ricardo's progress on the python-team
branch, I've almost fully rewritten the patch series and separated it in
two.

First part in 70855 is for patches that bring improvements that would
have been welcome even without the patch series, but that are a
prerequisite for the patch series.

Second part in 70858 contains the input removals.

Other patch series are superseded and will be closed.

>
> On 2024-04-22 16:40, Ricardo Wurmus wrote:
>
>>>  TL;DR :
>>>  - patch series in big progress, not done yet because I don't really
>>>  know where to stop and massive rebuilds.
>>
>> Please take a look at the python-team branch, which contains changes to
>> the build system.

-- 
Best regards,
Nicolas Graves

Re: Python's native-inputs

[Maxim Cournoyer]

Hi Nicolas,

Nicolas Graves <ngraves@ngraves.fr> writes:

[...]

>  TL;DR :
>  - patch series in big progress, not done yet because I don't really
>  know where to stop and massive rebuilds.
>  - WDYT about tweaking the build-system for pytest?
>  - not done : tweaking the pypi import to ignore those packages. I've
>  done it in the juliahub import code, but I haven't delved into that
>  though.
>  - not done : adding those packages to guix lint - trivial. We should
>  probably add a variable in pypi import that is exported and added into
>  guix lint.
>
>  Acting on the three (guix import, guix lint, guix build) can make it
>  quite easy on maintainers in the end IMO.

You're bringing good arguments.  If you can pull it off in the way
outlined above, I agree it's be valuable to have.

-- 
Thanks,
Maxim