From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id QPduO9eK0WB4bQEAgWs5BA (envelope-from ) for ; Tue, 22 Jun 2021 09:01:43 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id cGQ1N9eK0WAcUAAA1q6Kng (envelope-from ) for ; Tue, 22 Jun 2021 07:01:43 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1AF6217DC6 for ; Tue, 22 Jun 2021 09:01:43 +0200 (CEST) Received: from localhost ([::1]:50176 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvaPl-0005aP-Rf for larch@yhetil.org; Tue, 22 Jun 2021 03:01:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54378) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvaPD-0005aF-Mz for guix-devel@gnu.org; Tue, 22 Jun 2021 03:01:07 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:60050) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvaP7-0002LS-GW for guix-devel@gnu.org; Tue, 22 Jun 2021 03:01:07 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4G8HNq1NSDz1s6gn; Tue, 22 Jun 2021 09:00:59 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4G8HNq0nWmz1qqkQ; Tue, 22 Jun 2021 09:00:59 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id zZJNOq6Yg3wW; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-52-81.dynamic.mnet-online.de [188.174.52.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) Received: from thisbe.goebel-consult.de (hermia.goebel-consult.de [192.168.110.7]) by hermia.goebel-consult.de (Postfix) with ESMTP id 7DFAE600F9; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) Subject: Re: Questions regarding Python packaging To: Lars-Dominik Braun References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> From: Hartmut Goebel Organization: crazy-compilers.com Message-ID: <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> Date: Tue, 22 Jun 2021 09:00:54 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------78C137176FE39D2B54BEEB94" Content-Language: en-US Received-SPF: none client-ip=212.18.0.10; envelope-from=h.goebel@crazy-compilers.com; helo=mail-out.m-online.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, 46848@debbugs.gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624345303; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=UCW/eFCbrF0QBaGXmilcjjWTvcnZjRck+yw2ho8AJvM=; b=CpnfLVp8af6DZhY7CyRWnbQypVB5+Hh4LbRp0F8FRoAHbKV5K2UNrW3k0cT0eIxNVzWM1m NictzCyp6Gzh1p9UJ4AeKZbOwcVR99W9CTrGNfE9oWRquj3o6CV5kRw0UfpPVoabaj64fh z/H7t9vEoFhtb1+q0zlmRSFD8AZhTTFq0otin5fUmSd3CyzEg3Dk+3VO52r6fG9LvTsBwI SjqQZKzukzEN2/38hWBDhf08lWYE5AC5eX6+CbCIaceBkI0VE+MkigKL2+KYBIFyEGs2JY irHBvy0gF9Gyos5rmDaVDe62+Pfa1BW+cX8t3XXh+RFUpJludGPS8SnY0yBKpQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624345303; a=rsa-sha256; cv=none; b=YpVoDx/yr+6bj49ngwdBfv1AoQgip+wTtR/BrPeIq1Vq0NEfsEWwAmEqwzVH8xpDAxwU4T kIwL0IUGfxrKTc0b8W5RyzOFJEZiXnSGb1d+bMuOnyUMuny+vXaH5tAM5Zaf2XSTQ+sHbM hI8rccfmNvXIMjo0R8ca1WW59tKfLYSspQnWYtDjCkL94i5JUr+/d0E2Tc8cYBMm6qtpt6 t+mDby4ChM8yW9JBAPbjPZ+3HeXgjvsjKi+tllYBwtWe8Dfj/qJFfBt7h3vyR/0CKzKGzT FOjwtAAtIVB4c5Mm126mgVx/54jlQPPHGxfFX19KaF3sM4l4R2s3B6C17ZpVHg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.42 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 1AF6217DC6 X-Spam-Score: -2.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: ZKAqE2+3Wdo1 This is a multi-part message in MIME format. --------------78C137176FE39D2B54BEEB94 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi Lars, sorry for being late for commenting on this (the time I can spend on guix is rather limited atm). Here are some general remarks on this patch-set (in order of appearance): * Not installing pip by default might break some user's environments. Anyhow, since using pip in guix is not such a good idea anyway, this should be okay. * "use-setuptools" is gone. There are still about 10 packages with "#:use-setuptools #f" - which means they are (expected to be) incompatible with setuptools for some reason. You might want to check whether these packages actually still can't be packages with setuptools. * setuptools-shim has been removed. I don't think this is a good idea, since this peace of code enforces packages to be actually build with setuptools instead of old distutils. This code is still in current pip, so I assume it is still required. (This shim ensures setuptools is used, even if setup.py only imports distutils. And setuptools is required for some options like ""--single-version-externally-managed" - as the comment for the shim says.) * set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much more helpful than the new one-line comment. * set-SOURCE-DATE-EPOCH: This implementation makes the code depend on wheel and wheel being used for installation. * Why has rename-pth-file been removed? Are you sure .pth-files are never created anymore nowerdays? * python-hashbang: Isn't this done already by the normal "patch-shebangs" phase after install in  gnu-build-system? (BTW: these are called *she*bangs). * I suggest to have phase compile-bytecode still honor older versions of python > 1) Validate the general idea of using pypa-build is viable and > sustainable in the long run – ideally through review by someone else > than me. We can’t touch python-build-system every week to solve > structural issues, so it needs to be bullet-proof. pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as far as I can see, adoption is low. > 2) Figure out how to run testing code. Currently python-build-system > just picks pytest, if available – not sure this is the best option we > have. How do we deal with other test systems? How do we pass options? AFAIK fhere is no standard way for running tests in python. pytest seems to be the most modern test-system. Anyhow packages still use nose or tox (which again might run pytest or nose, with parameters fetched from tox.ini). So I'm afraid, there is no general rule. Did the PyPA publish some recommendations or PEP on this? > 4) Iron out minor details like including pip in the python package or > create a new python-toolchain package? What do we include in that > meta-package? pip? virtualenv? …? As I Python developer I nowerdays would expect pip and venv (which is part of the std-lib - but not the virualenv, which is a separate module) to be availalbe when installing "python". Anyhow I could live with pip being a separate package. "python-toolchain" sounds oversized for me. Would this include the C-compiler, too (which one? maybe I want to build cross). I'd rather not have such a package. > 5) Fix my awkward Scheme code, especially regarding unpacking of the > built wheels. Should we be using Python’s unzip module or can be > assumed unzip is available in the build environment? (Should we add > it?) The gnu-build-system already provides the "unzip" binary (used in phase "unpack"). So we could simply use this. Otherwise I recommend using the Python zip module, as this is what is used for creating the zip-archives :-) -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------78C137176FE39D2B54BEEB94 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
Hi Lars,

sorry for being late for commenting on this (the time I can spend on guix is rather limited atm).

Here are some general remarks on this patch-set (in order of appearance):

  • Not installing pip by default might break some user's environments. Anyhow, since using pip in guix is not such a good idea anyway, this should be okay.

  • "use-setuptools" is gone. There are still about 10 packages with "#:use-setuptools #f" - which means they are (expected to be) incompatible with setuptools for some reason. You might want to check whether these packages actually still can't be packages with setuptools.

  • setuptools-shim has been removed. I don't think this is a good idea, since this peace of code enforces packages to be actually build with setuptools instead of old distutils. This code is still in current pip, so I assume it is still required.

(This shim ensures setuptools is used, even if setup.py only imports distutils. And setuptools is required for some options like ""--single-version-externally-managed" - as the comment for the shim says.)

  • set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much more helpful than the new one-line comment.

  • set-SOURCE-DATE-EPOCH: This implementation makes the code depend on wheel and wheel being used for installation.

  • Why has rename-pth-file been removed? Are you sure .pth-files are never created anymore nowerdays?

  • python-hashbang: Isn't this done already by the normal "patch-shebangs" phase after install in  gnu-build-system? (BTW: these are called *she*bangs).

  • I suggest to have phase compile-bytecode still honor older versions of python 


1) Validate the general idea of using pypa-build is viable and
   sustainable in the long run – ideally through review by someone else
   than me. We can’t touch python-build-system every week to solve
   structural issues, so it needs to be bullet-proof.

pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as far as I can see, adoption is low.

2) Figure out how to run testing code. Currently python-build-system
   just picks pytest, if available – not sure this is the best option we
   have. How do we deal with other test systems? How do we pass options?

AFAIK fhere is no standard way for running tests in python. pytest seems to be the most modern test-system. Anyhow packages still use nose or tox (which again might run pytest or nose, with parameters fetched from tox.ini). So I'm afraid, there is no general rule.

Did the PyPA publish some recommendations or PEP on this?

4) Iron out minor details like including pip in the python package or
   create a new python-toolchain package? What do we include in that
   meta-package? pip? virtualenv? …?

As I Python developer I nowerdays would expect pip and venv (which is part of the std-lib - but not the virualenv, which is a separate module) to be availalbe when installing "python". Anyhow I could live with pip being a separate package.

"python-toolchain" sounds oversized for me. Would this include the C-compiler, too (which one? maybe I want to build cross). I'd rather not have such a package. 

5) Fix my awkward Scheme code, especially regarding unpacking of the
   built wheels. Should we be using Python’s unzip module or can be
   assumed unzip is available in the build environment? (Should we add
   it?)
The gnu-build-system already provides the "unzip" binary (used in phase "unpack"). So we could simply use this. Otherwise I recommend using the Python zip module, as this is what is used for creating the zip-archives :-) 
-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |
--------------78C137176FE39D2B54BEEB94--