From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YIoCLNOfHmKEJAAAgWs5BA (envelope-from ) for ; Tue, 01 Mar 2022 23:36:03 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id iFmlKNOfHmK0AQEAauVa8A (envelope-from ) for ; Tue, 01 Mar 2022 23:36:03 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C994715FC2 for ; Tue, 1 Mar 2022 23:36:02 +0100 (CET) Received: from localhost ([::1]:49046 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPB69-0002rw-Tw for larch@yhetil.org; Tue, 01 Mar 2022 17:36:01 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPAg3-0007A1-Ln for guix-devel@gnu.org; Tue, 01 Mar 2022 17:09:03 -0500 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:50251) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPAfw-0006Yh-Ew for guix-devel@gnu.org; Tue, 01 Mar 2022 17:09:03 -0500 Received: (Authenticated sender: tanguy@bioneland.org) by mail.gandi.net (Postfix) with ESMTPSA id E2FD81BF203; Tue, 1 Mar 2022 22:08:51 +0000 (UTC) Content-Type: multipart/mixed; boundary="===============5605966078630222987==" MIME-Version: 1.0 References: <20220301153619.096d56c8@tachikoma.lepiller.eu> Subject: Re: Creating subtitles for the Guix Days videos! From: Tanguy LE CARROUR To: Julien Lepiller In-Reply-To: <20220301153619.096d56c8@tachikoma.lepiller.eu> Date: Tue, 01 Mar 2022 23:08:50 +0100 Message-ID: <164617253081.21060.6792004916228135286@localhost> User-Agent: alot/0.10 Received-SPF: none client-ip=217.70.183.201; envelope-from=tanguy@bioneland.org; helo=relay8-d.mail.gandi.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1646174163; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=LO8aW6slWqzwQn+FridzbwCM1ETvJ+oRYe/gB2Lou0k=; b=MZ2qVPZIlbWwS7bpQY3ZecUEvbzeiVc7W5tUWkQaRTSaKL7WPaYcKN+F7HTq+vrTEvVWyU nHJXrC9MpI7wJ3kg1TNxKzXrdrSuoT1M3YEbd4O42erA2FLl6JO7Pow4E6TK+gIsjOlpxw 3oOuUkDPEr99tnzpImob9xtgC7E5j8/8V83InVb0X8hAWtafD8V4TPeH5XsV2VpLsYAhFS bd/mp/Fn7V/8jV1JNtGA1fD1m3tBbIrhuMrYblhsVYYmxfAAwNfc7lsQCU3wNdc3aL1vQ1 wgxISLoO72jhZrJ8wW86VOP1BwTV5qZ2abZ0zj+iRMsEzp68NkyEC5re2js5UQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1646174163; a=rsa-sha256; cv=none; b=B8R18Y9e7xXxZnTs1EulLgjfAYO9OkPuF4vKKCv6njn0cYZ0XHWT0ouk2I5vwI4ZQYFXpe PfXD+4zfMToUv+DI+paDDyjxaGZpVr1KJ80zrwtqcHr1x0gNAE/j9zAc7JHs4ERQRigPlF zO+FkGMZB3/d8uzzfN7LCARQAWTiika9Pzr9zwW1hD5qQBOISWAQy62/GwcsN69XelPwz4 3AquOCH+96BrHY0/MZhcJJiVfZR2DgYmOCveZZ31v/yoBnGpNvHXRX8xEFEoam5ohHhV5G itxwQxNxkkrE9giZ8NTvnJr6rOcPiu6o104xyfPdSJOLwILtUr/zYcVeK+3X2w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.08 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: C994715FC2 X-Spam-Score: -3.08 X-Migadu-Scanner: scn0.migadu.com X-TUID: trb4fW8Vnk8l --===============5605966078630222987== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Julien, Quoting Julien Lepiller (2022-03-01 15:36:19) > I'm looking for volunteers to create English subtitles for the Guix Days > talks. [=E2=80=A6] Please send me the subtitles once > they are completed, I'll add them with the videos. It's my first time, so thank you for your indulgence! :-) I'm attaching my humble contribution: - `.txt` the transcription=C2=A0; - `.ass` the file created by Aegisub=C2=A0; and - `.sub` an attempt to export it to sub format. I have to admit that is pretty bad "punctuation-wise", because I was not sure were the sentences started and ended. Sorry! Just let me know if I have to fix anything. Regards, --=20 Tanguy --===============5605966078630222987== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="guix-days-2022-modernizing-python-build-system.mkv.txt"; maxlinelen="78" hello everyone nice to have you here at Guix Days 2022 I would like to talk about Python build system and why we need to modernize it python-build-system is the Guix component that turns the source distribution of any Python package out there into an installable image so, basically, a directory under the GNU store let's have a look at tomli a quite simple Python package which has no external dependencies so, if we import it using `guix import`, here add a few imports to the resulting file and then try to build it it should work out of the box, right? The problem is, it does not! and instead we see at the bottom an error message saying "no setup.py found" and, indeed, if we look at the source distribution there is no `setup.py` to be found anywhere So, what is going on? Fortunately, this package is already available in Guix as `python-tomli` so we can have a look at its definition to see how it is currently built let's just do that looking at the build system's arguments we see the phases `build` here and `install` here which are usually provided by Python build system replaced with custom code I'm only showing the interesting parts here the actual commands are actually much longer first the build phase uses a Python module called `build` to build the wheel, as we can see here the wheel is basically a distribution format in the Python world then in the install phase we simply use a well known tool called Pip to install the wheel that we just built into the output which would be somewhere around the GNU store so how does the build module knows what to do what to build? it follows PEP 517 PEPs are kind of the RFCs of the Python world and this PEP basically splits building wheels into two parts a frontend and a backend the frontend is the user facing part for example the `build` we just saw here this is the user facing part of the build process and then a backend the frontend is supposed to read a file called `pyproject.toml` this is what we are seeing here and in that TOML file, a section called `build-system` this one here declares which backend will actually build our wheel and, in this case, another package called `flit_core` its requirements a build time dependency of tomli and its module `flit_core.buildapi` is providing us with the build entrypoint. The file also contains standardize metadata and tool related configuration data which I'm not showing here A PEP 517* compatible build backend provides a standard function entrypoint called `build_wheel` in the module I just referenced here in the top and, if we call it, it will just do its magic and it will produce a wheel file and its first argument it's the wheel directory that wheel is basically a zip file with a predefined structure that we can extract into our store and we are almost done and this is what Pip does in the install phase here and that's basically the entire build process as specified by PEP 517 there is no `setup.py` involved any more we don't have to call it we don't have to create it as a package provider so the reason why the error message I showed, showed up earlier will keep on poping up more and more is simple: we are late! we are really really late, actually because PEP=C2=A0517 was originally created in 2015 and that it gained provisional acceptance in 2017 and just last year, after being basically being the *de facto* successor of `setup.py` for some= time it has been finalized and fully accepted and more importantly, flit which you remember from the previous slide is also able to create source distributions and upload them to PyPI Python public package repository basically so far, it has been generating a `setup.py` and does nobody really noticed but since version 3.5, which was released in November 2021 flit stop doing that by default and thus we are seeing more and more packages without `setup.py` in their source distributions and so we are basically unable to build this projects right now or this Python modules a look at the Guix's repository in late January and back then only 11 packages actually used Pip or PyPA build as we've seen but I think our ecosystem is quite old about half the packages not being the latest versions available upstream according to `guix refresh` so it's possible that more packages actually require support for this `pypr= oject.toml` and we simply have not updated them yet for whatever reason maybe because it's too difficult or nobody poped to do it yet they are also more issues with our current Python build system for instance `setup.py` test ???? has been deprecated for quite some time since 2019 actually and the Python build system's default check phase relies on that even though it doesn't work for a lot of packages right now and thus, almost every package in our repository needs to replace that check phase with a call to Pytest which is the *de facto* standard right now for executing tests ???? tox does something similar but not quite the same another long standing issue is wrapping of Python path or Guix Python path now because it includes native inputs and thus propagates way too many packages and I'm sure we can find more issues with the current Python build system thus, for almost a year now, I've been working on a modernized Python build= system that addresses some of this issues unfortunately I haven't had much time lately to improve it for others I hope to pick it up again and get it into shape and get it to master at some point you can read about it here in Guix issue #46848 or you can just pull a branch called `wip-python-pep517` it's also build by the CI by the way so you don't have to do the whole world rebuild by yourself and it's using exactly the method I described above the part written in Guile ends up reading the project's `pyproject.toml` and then calls the build system's entrypoint `build_wheel` as we can see he= re and if there's no `pyproject.toml` it's also not a problem actually because setuptools, which is still provided by default, and still included in the build system by default, because setuptools still provides a PEP 517 build backend which simply uses the existing `setup.py` behind the curtains so we don't have to, like provide extra logic for that they were actually other options for this build phase in how to do it basically for example, we could also use the build module that we have seen above and some packages use currently however that raises questions about bootstraping how do you build the build package if it essentially requires itself because the build module is part of the Python build system and of of it's dependency tomli for example is one of the dependency of build and what do we do with tomli like how do we install it without having build so this is tricky, and I tried it in the beginning and it works but we need to do ???? manual copying and that's not really pretty to be honest we could also use Pip which is usually distributed with Python also our python package bundles Pip and setuptools this has caused a few issues in the past because we cannot update either of them for provide alternative newer versions without having a full world rebuild basically and also Pip being a massive project it bundles quite a few of it's dependencies and I feel "we" as a Guix project should try to untangle that mess whenever we can and in this case, we actually can so, in conclusion, I think running this 4 lines of Python code that you can see here is a much simpler solution that trying to bootstrap the build module, tomli and all the modules required to get a Python based builder basically we can just do it in 4 lines of code there are a few changes for packagers which aime to reduce the need for custom phases first, it is possible to override the build backend detected from the `pypr= oject.toml` especially early adopters of PEP 517 use modules which have been renamed in the meantime I think both Poetry and Flit did that at some point the split their build logic into a separate project and renamed the backend so, for some packages, it's necessary to override this detected build backe= nd and you can do that simply by specifying the `#:build-backend` option and also if the detection is wrong at some point we could override it without causing too much damage secondly, we have `#:configure-flags` they had existed before, but where applied only during install but not during the build phase which, I believe, is a ????, we'll see and this are arguments to `setup.py` but their syntax depends on the build backend unfortunately sometimes you can pass the argument directly sometimes you need to pass them as an option to another option like we can see here that's the setuptools specific way to do it so you have an option and then you have to pass the actual options as an argument to that option and, in theory, they could also look completly different like no relationship to any command line flags it's just legacy code right now and it's not pretty and it's one of the things that have been criticized about PEP=C2=A0517 this configure flags are not really specified how they are supposed to look but this solution exists, so we have to swim with the flow, kind of and, finally, they are `#:test-flags` my proposal was to auto detect test binary based on the package's native in= puts so if pytest is available, it will just use pytest if nose is available, it will use nose and so on now, test flags override the default arguments passed to this test command and, hopefully, this can reduce the number of custom phases required to exclude tests mostly in this case, we are simply don't want to run the `test_legacy_callbacks` f= unction and so we can just exclude it with the flags and the package will build just fine, probably, hopefully so, where do we go from here? the first step would be to finalize the work on the Python build system and get the branch ready into a state where it can be merged unfortunately, right now, quite a few packages fail to build there because their previously disabled test suite fails not necessarily disabled, but it was never ran because the test runner was buggy and the `setup.py test` command simply di= d nothing and didn't detect edge cases we could also go a different route and add a new build system without touching the old one and just slowly migrate every Python package we have over instead of doing the big switch at once that was also one of the suggestions that was certainly possible I haven't explored that it would works as I mentioned earlier, currently the native inputs end up in wrapped binar= ies and I think this is one thing we should address while doing the whole world= rebuild anyway but I haven't done any work on that yet, unfortunately and finally, the pypi import needs support to read `pyproject.toml` not so much for the runtime dependencies, this is fine, we have another metadata file which contains the runtime depe= ndencies but for build time and test dependencies because they are not included in the wheel file, there's a file called metadata, and it doesn't have this data so right now, if you import a `pyproject.toml` based project you will lack the build dependencies basically for example flit will not be in the native input right now and that's it for me for now I hope to answer some of your questions during the live Q and A and in the PDF version of this slides, you'll find some clickable links to different talks and to the standards I have referenced so, I hope I'll see you soon! --===============5605966078630222987== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="guix-days-2022-modernizing-python-build-system.mkv.sub"; maxlinelen="78" =EF=BB=BF{29}{43}hello everyone {44}{109}nice to have you here at Guix Days 2022 {110}{161}I would like to talk about Python build system {162}{202}and why we need to modernize it {203}{251}python-build-system is the Guix component {252}{318}that turns the source distribution of any Python package out there {319}{347}into an installable image {348}{405}so, basically, a directory under the GNU store {406}{492}let's have a look at tomli a quite simple Python package {493}{526}which has no external dependencies {527}{608}so, if we import it using `guix import`, here {609}{668}add a few imports to the resulting file {669}{697}and then try to build it {698}{753}it should work out of the box, right? {754}{782}The problem is, it does not! {783}{840}and instead we see at the bottom {841}{927}an error message saying "no setup.py found" {928}{985}and, indeed, if we look at the source distribution {986}{1043}there is no `setup.py` to be found anywhere {1044}{1074}So, what is going on? {1075}{1142}Fortunately, this package is already available in Guix {1143}{1171}as `python-tomli` {1172}{1209}so we can have a look at its definition {1210}{1239}to see how it is currently built {1240}{1268}let's just do that {1269}{1327}looking at the build system's arguments {1328}{1395}we see the phases `build` here and `install` here {1421}{1486}which are usually provided by Python build system {1487}{1531}replaced with custom code {1532}{1582}I'm only showing the interesting parts here {1583}{1634}the actual commands are actually much longer {1635}{1681}first the build phase {1682}{1739}uses a Python module called `build` {1740}{1801}to build the wheel, as we can see here {1802}{1926}the wheel is basically a distribution format in the Python world {1927}{1995}then in the install phase {1996}{2052}we simply use a well known tool called Pip {2053}{2116}to install the wheel that we just built {2117}{2203}into the output which would be somewhere around the GNU store {2204}{2261}so how does the build module knows what to do {2262}{2291}what to build? {2292}{2355}it follows PEP 517 {2356}{2406}PEPs are kind of the RFCs of the Python world {2407}{2522}and this PEP basically splits building wheels into two parts {2523}{2551}a frontend and a backend {2552}{2585}the frontend is the user facing part {2586}{2638}for example the `build` we just saw here {2639}{2696}this is the user facing part of the build process {2697}{2725}and then a backend {2726}{2870}the frontend is supposed to read a file called `pyproject.toml` {2871}{2899}this is what we are seeing here {2900}{2993}and in that TOML file, a section called `build-system` {2994}{3015}this one here {3016}{3075}declares which backend will actually build our wheel {3076}{3160}and, in this case, another package called `flit_core` {3161}{3247}its requirements a build time dependency of tomli {3248}{3305}and its module `flit_core.buildapi` {3306}{3361}is providing us with the build entrypoints. {3362}{3421}The file also contains standardize metadata {3422}{3460}and tool related configuration data {3461}{3489}which I'm not showing here {3490}{3595}A PEP 517* compatible build backend {3596}{3653}provides a standard function entrypoint {3654}{3682}called `build_wheel` {3683}{3740}in the module I just referenced here in the top {3741}{3802}and, if we call it, it will just do its magic {3803}{3847}and it will produce a wheel file {3848}{3911}and its first argument it's the wheel directory {3912}{3968}that wheel is basically a zip file {3969}{3990}with a predefined structure {3991}{4032}that we can extract into our store {4033}{4081}and we are almost done {4082}{4146}and this is what Pip does in the install phase here {4234}{4291}and that's basically the entire build process {4292}{4326}as specified by PEP 517 {4327}{4378}there is no `setup.py` involved any more {4379}{4387}we don't have to call it {4388}{4465}we don't have to create it as a package provider {4491}{4575}so the reason why the error message I showed, showed up earlier {4576}{4610}will keep on poping up more and more {4611}{4645}is pretty simple we are late! {4646}{4697}we are really really late, actually {4698}{4784}because PEP=C2=A0517 was originally created in 2015 {4785}{4869}and that it gained provisional acceptance in 2017 {4870}{4898}and just last year, {4899}{4987}after being basically being the *de facto* successor of `setup.= py` for some time {4988}{5045}it has been finalized and fully accepted {5046}{5137}and more importantly, flit which you remember from the previous= slide {5138}{5197}is also able to create source distributions {5198}{5233}and upload them to PyPI {5234}{5297}Python's public package repository basically {5298}{5363}so far, it has been generating a `setup.py` {5364}{5393}and does nobody really noticed {5394}{5500}but since version 3.5, which was released in November 2021 {5501}{5561}flit stop doing that by default {5562}{5627}and thus we are seeing more and more packages without `setup.py` {5628}{5656}in their source distributions {5657}{5734}and so we are basically unable to build this projects right now {5735}{5763}or this Python modules {5764}{5830}a look at the Guix's repository in late January {5831}{5896}and back then only 11 packages actually used Pip {5897}{5939}or PyPA build as we've seen {5940}{5996}but I think our ecosystem is quite old {5997}{6060}with about half the packages not being the latest versions avai= lable upstream {6061}{6089}according to `guix refresh` {6090}{6193}so it's possible that more packages actually require support fo= r this `pyproject.toml` {6194}{6263}and we simply have not updated them yet for whatever reason {6264}{6321}maybe because it's too difficult or nobody poped to do it yet {6351}{6437}they are also more issues with our current Python build system {6438}{6546}for instance `setup.py` test ???? has been deprecated for quite= some time {6547}{6589}since 2019 actually {6590}{6675}and the default Python build system's default check phase relie= s on that {6676}{6727}even though it doesn't work for a lot of packages right now {6728}{6791}and thus, almost every package in our repository {6792}{6872}needs to replace that check phase with a call to Pytest {6873}{6930}which is the *de facto* standard right now for executing tests {6931}{7021}???? tox does something similar but not really quite the same {7022}{7104}another long standing issue is wrapping of Python path {7105}{7133}or Guix Python path now {7134}{7178}because it includes native inputs {7179}{7222}and thus propagates way too many packages {7223}{7307}and I'm sure we can find more issues with the current Python bu= ild system {7308}{7481}thus, for almost a year now, I've been working on a modernized = Python build system {7482}{7513}that addresses some of this issues {7514}{7588}unfortunately I haven't had much time lately to improve it for = other {7589}{7615}I hope to pick it up again {7616}{7628}and get it into shape {7629}{7657}and get it to master at some point {7658}{7686}you can read about it here {7687}{7771}in Guix issue #46848 {7772}{7858}or you can just pull a branch called `wip-python-pep517` {7859}{7904}it's also build by the CI by the way {7905}{7966}so you don't have to do the whole world rebuild by yourself {7967}{8032}and it's using exactly the method I described above {8034}{8148}the part written in Guile ends up reading the project's `pyproj= ect.toml` {8149}{8264}and then calls the build system's entrypoint `build_wheel` as w= e can see here {8265}{8351}and if there's no `pyproject.toml` it's also not a problem actu= ally {8352}{8438}because setuptools, which is still provided by default, {8439}{8473}and still included in the build system by default, {8474}{8564}because setuptools provides a PEP 517 build backend {8591}{8673}which simply uses the existing `setup.py` behind the curtains {8674}{8757}so we don't have to, like provide extra logic for that {8787}{8847}they were actually other options for this build phase {8848}{8876}in how to do it basically {8903}{8943}for example, we could also use the build module {8944}{8963}that we have seen above {8964}{9016}and some packages use currently {9017}{9063}however that raises questions about bootstraping, like {9064}{9134}how do you build the build package if it essentially requires i= tself {9135}{9184}because the build module is part of the Python build system {9185}{9221}and of of it's dependency {9222}{9279}tomli for example is one of the dependency of build {9280}{9337}and what do we do with tomli {9338}{9367}like how do we install it without having build {9368}{9434}so this is tricky, and I tried it in the beginning {9435}{9457}and it works {9458}{9521}but we need to do like some weird manual copying {9522}{9569}and that's not really pretty to be honest {9570}{9598}we could also use Pip {9599}{9656}which is usually distributed with Python {9657}{9721}also our python package bundles Pip and setuptools {9722}{9772}this has caused a few issues in the past {9773}{9801}because we cannot update either of them {9802}{9853}for provide alternative newer versions {9854}{9917}without having a full world rebuild basically {9918}{9959}and also Pip being a massive project {9960}{10004}it bundles quite a few of it's dependencies {10005}{10033}and I feel "we" as a Guix project {10034}{10091}should try to untangle that mess whenever we can {10092}{10120}and in this case, we actually can {10121}{10207}so, in conclusion, I think running this 4 lines of Python code {10208}{10236}that you can see here {10262}{10294}is a much simpler solution {10295}{10351}than trying to bootstrap the build module, {10352}{10468}and tomli and all the modules required to get a Python based = builder basically {10469}{10501}we can just do it in 4 lines of code {10556}{10606}there are a few changes for packagers {10607}{10654}which aime to reduce the need for custom phases {10655}{10787}first, it is possible to override the build backend detected = from the `pyproject.toml` {10788}{10848}especially early adopters of PEP 517 {10849}{10905}use modules which have been renamed in the meantime {10933}{10990}I think both Poetry and Flit did that at some point {10991}{11048}and they split their build logic into a separate project {11049}{11077}and renamed the backend {11078}{11164}so, for some packages, it's necessary to override this detect= ed build backend {11165}{11222}and you can do that simply by specifying the `#:build-backend= ` option {11223}{11280}and also if the detection is wrong at some point {11281}{11338}we can just override it without causing too much damage {11345}{11416}secondly, we have `#:configure-flags` {11425}{11483}they have existed before, but where applied only during insta= ll {11484}{11512}but not during the build phase {11513}{11552}which, I believe, is a ????, we'll see {11553}{11628}and this are kind of arguments to `setup.py` {11629}{11676}but their syntax depends on the build backend unfortunately {11677}{11738}so sometimes you can pass the arguments directly {11739}{11788}sometimes you need to pass them as an option to another option {11789}{11813}like we can see here {11814}{11870}I think that's the setuptools specific way to do it {11871}{11906}so you have an option {11907}{11976}and then you have to pass the actual option as an argument to= that option {11977}{12063}and, in theory, they could also look completly different {12064}{12120}like with no relationship to any command line flag {12121}{12181}it's just legacy code/stuff right now {12182}{12208}and it's not pretty {12209}{12271}and it's one of the things that have been criticized about PE= P=C2=A0517 {12272}{12353}this configure flags are not really specified how they are su= pposed to look {12354}{12440}but this solution exists, so we have to swim with the flow, k= ind of {12444}{12500}and, finally, they are `#:test-flags` {12501}{12614}my proposal was to auto detect test binary based on the packa= ge's native inputs {12615}{12673}so if pytest is available, it will just use pytest {12674}{12723}if nose is available, it will use nose {12724}{12737}and so on {12738}{12826}now, test flags override the default arguments passed to this= test command {12827}{12892}and, hopefully, this can reduce the number of custom phases r= equired {12893}{12925}to exclude tests mostly {12926}{13015}in this case, we are simply don't want to run the `test_legac= y_callbacks` function {13016}{13064}and so we can just exclude it with the flags {13092}{13157}and the package will build just fine, probably, hopefully {13158}{13220}so, where do we go from here? {13221}{13310}the first step would be to finalize the work on the Python bu= ild system {13311}{13404}and get the branch ready into a state where it can be merged {13405}{13484}unfortunately, right now, quite a few packages fail to build = there {13485}{13571}because their previously disabled test suite fails {13572}{13612}not necessarily disabled, but it was never ran {13613}{13713}because the test runner was buggy and the `setup.py test` com= mand simply did nothing {13714}{13745}and didn't detect edge cases {13746}{13816}we could also go a different route and add a new build system {13817}{13841}without touching the old one {13842}{13913}and just slowly migrate every Python package we have over {13914}{13945}instead of doing the big switch at once {13949}{13988}that was also one of the suggestions that was certainly possi= ble {13989}{14027}I have explored that it would work {14028}{14154}as I mentioned earlier, currently the native inputs end up in= wrapped binaries {14155}{14280}and I think this is one thing we should address too while doi= ng the whole world rebuild anyway {14281}{14354}but I haven't done any work on that yet, unfortunately {14355}{14444}and finally, the pypi import needs support to read `pyproject= .toml` {14445}{14499}not so much for the runtime dependencies, {14500}{14582}this is fine, we have another metadata file which contains th= e runtime dependencies {14583}{14615}but for build time and test dependencies {14616}{14668}because they are not included in the wheel file, {14669}{14754}there's a file called metadata, and it doesn't have this data {14755}{14870}so right now, if you import a `pyproject.toml` based project {14871}{14934}you will lack the build dependencies basically {14935}{15020}for example flit will not be in the native inpust right now {15021}{15091}and that's it for me for now {15092}{15166}I hope to answer some of your questions during the live Q and= A {15167}{15253}and in the PDF version of this slides, you'll find some click= able links bellow {15254}{15340}to different talks and to the standards I have referenced {15341}{15369}so, I hope I'll see you soon! --===============5605966078630222987== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="guix-days-2022-modernizing-python-build-system.mkv.ass"; maxlinelen="78" =EF=BB=BF[Script Info] ; Script generated by Aegisub 3.2.2 ; http://www.aegisub.org/ Title: Default Aegisub file ScriptType: v4.00+ WrapStyle: 0 ScaledBorderAndShadow: yes YCbCr Matrix: TV.601 PlayResX: 1916 PlayResY: 1076 [Aegisub Project Garbage] Export Encoding: Unicode (UTF-8) Audio File: guix-days-2022-modernizing-python-build-system.mkv Video File: guix-days-2022-modernizing-python-build-system.mkv Video AR Mode: 4 Video AR Value: 1.780669 Video Zoom Percent: 0.250000 Active Line: 248 Video Position: 15341 [V4+ Styles] Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineCo= lour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spaci= ng, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, Margi= nV, Encoding Style: Default,Arial,20,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,0,0,0,0= ,100,100,0,0,1,2,2,2,10,10,10,1 [Events] Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, = Text Dialogue: 0,0:00:01.64,0:00:02.60,Default,,0,0,0,,hello everyone Dialogue: 0,0:00:02.60,0:00:06.92,Default,,0,0,0,,nice to have you here at = Guix Days 2022 Dialogue: 0,0:00:06.92,0:00:10.64,Default,,0,0,0,,I would like to talk abou= t Python build system Dialogue: 0,0:00:10.64,0:00:13.32,Default,,0,0,0,,and why we need to modern= ize it Dialogue: 0,0:00:13.32,0:00:16.76,Default,,0,0,0,,python-build-system is th= e Guix component Dialogue: 0,0:00:16.76,0:00:21.38,Default,,0,0,0,,that turns the source dis= tribution of any Python package out there Dialogue: 0,0:00:21.38,0:00:23.38,Default,,0,0,0,,into an installable image Dialogue: 0,0:00:23.38,0:00:27.78,Default,,0,0,0,,so, basically, a director= y under the GNU store Dialogue: 0,0:00:27.78,0:00:33.14,Default,,0,0,0,,let's have a look at toml= i a quite simple Python package Dialogue: 0,0:00:33.14,0:00:36.08,Default,,0,0,0,,which has no external dep= endencies Dialogue: 0,0:00:36.08,0:00:41.12,Default,,0,0,0,,so, if we import it using= `guix import`, here Dialogue: 0,0:00:41.12,0:00:45.94,Default,,0,0,0,,add a few imports to the = resulting file Dialogue: 0,0:00:45.94,0:00:47.94,Default,,0,0,0,,and then try to build it Dialogue: 0,0:00:47.94,0:00:51.12,Default,,0,0,0,,it should work out of the= box, right? Dialogue: 0,0:00:51.12,0:00:53.84,Default,,0,0,0,,The problem is, it does n= ot! Dialogue: 0,0:00:53.84,0:00:57.38,Default,,0,0,0,,and instead we see at the= bottom Dialogue: 0,0:00:57.38,0:01:03.20,Default,,0,0,0,,an error message saying "= no setup.py found" Dialogue: 0,0:01:03.20,0:01:07.64,Default,,0,0,0,,and, indeed, if we look a= t the source distribution Dialogue: 0,0:01:07.64,0:01:11.32,Default,,0,0,0,,there is no `setup.py` to= be found anywhere Dialogue: 0,0:01:11.32,0:01:13.86,Default,,0,0,0,,So, what is going on? Dialogue: 0,0:01:13.86,0:01:18.26,Default,,0,0,0,,Fortunately, this package= is already available in Guix Dialogue: 0,0:01:18.26,0:01:20.26,Default,,0,0,0,,as `python-tomli` Dialogue: 0,0:01:20.26,0:01:22.64,Default,,0,0,0,,so we can have a look at = its definition Dialogue: 0,0:01:22.64,0:01:24.64,Default,,0,0,0,,to see how it is currentl= y built Dialogue: 0,0:01:24.64,0:01:26.64,Default,,0,0,0,,let's just do that Dialogue: 0,0:01:26.64,0:01:30.68,Default,,0,0,0,,looking at the build syst= em's arguments Dialogue: 0,0:01:30.68,0:01:35.92,Default,,0,0,0,,we see the phases `build`= here and `install` here Dialogue: 0,0:01:37.36,0:01:42.04,Default,,0,0,0,,which are usually provide= d by Python build system Dialogue: 0,0:01:42.04,0:01:44.66,Default,,0,0,0,,replaced with custom code Dialogue: 0,0:01:44.66,0:01:48.36,Default,,0,0,0,,I'm only showing the inte= resting parts here Dialogue: 0,0:01:48.36,0:01:52.10,Default,,0,0,0,,the actual commands are a= ctually much longer Dialogue: 0,0:01:52.10,0:01:55.50,Default,,0,0,0,,first the build phase Dialogue: 0,0:01:55.50,0:01:59.60,Default,,0,0,0,,uses a Python module call= ed `build` Dialogue: 0,0:01:59.60,0:02:03.80,Default,,0,0,0,,to build the wheel, as we= can see here Dialogue: 0,0:02:03.80,0:02:12.14,Default,,0,0,0,,the wheel is basically a = distribution format in the Python world Dialogue: 0,0:02:12.14,0:02:16.54,Default,,0,0,0,,then in the install phase Dialogue: 0,0:02:16.54,0:02:20.50,Default,,0,0,0,,we simply use a well know= n tool called Pip Dialogue: 0,0:02:20.50,0:02:25.06,Default,,0,0,0,,to install the wheel that= we just built Dialogue: 0,0:02:25.06,0:02:31.04,Default,,0,0,0,,into the output which wou= ld be somewhere around the GNU store Dialogue: 0,0:02:31.04,0:02:35.58,Default,,0,0,0,,so how does the build mod= ule knows what to do Dialogue: 0,0:02:35.58,0:02:37.58,Default,,0,0,0,,what to build? Dialogue: 0,0:02:37.58,0:02:41.84,Default,,0,0,0,,it follows PEP 517 Dialogue: 0,0:02:41.84,0:02:45.48,Default,,0,0,0,,PEPs are kind of the RFCs= of the Python world Dialogue: 0,0:02:45.48,0:02:52.78,Default,,0,0,0,,and this PEP basically sp= lits building wheels into two parts Dialogue: 0,0:02:52.78,0:02:54.78,Default,,0,0,0,,a frontend and a backend Dialogue: 0,0:02:54.78,0:02:57.72,Default,,0,0,0,,the frontend is the user = facing part Dialogue: 0,0:02:57.72,0:03:00.94,Default,,0,0,0,,for example the `build` w= e just saw here Dialogue: 0,0:03:00.94,0:03:04.78,Default,,0,0,0,,this is the user facing p= art of the build process Dialogue: 0,0:03:04.78,0:03:06.78,Default,,0,0,0,,and then a backend Dialogue: 0,0:03:06.78,0:03:16.84,Default,,0,0,0,,the frontend is supposed = to read a file called `pyproject.toml` Dialogue: 0,0:03:16.84,0:03:18.84,Default,,0,0,0,,this is what we are seein= g here Dialogue: 0,0:03:18.84,0:03:25.70,Default,,0,0,0,,and in that TOML file, a = section called `build-system` Dialogue: 0,0:03:25.70,0:03:27.30,Default,,0,0,0,,this one here Dialogue: 0,0:03:27.30,0:03:31.48,Default,,0,0,0,,declares which backend wi= ll actually build our wheel Dialogue: 0,0:03:31.48,0:03:36.98,Default,,0,0,0,,and, in this case, anothe= r package called `flit_core` Dialogue: 0,0:03:36.98,0:03:42.68,Default,,0,0,0,,its requirements a build = time dependency of tomli Dialogue: 0,0:03:42.68,0:03:46.96,Default,,0,0,0,,and its module `flit_core= .buildapi` Dialogue: 0,0:03:46.96,0:03:50.46,Default,,0,0,0,,is providing us with the = build entrypoints. Dialogue: 0,0:03:50.46,0:03:55.12,Default,,0,0,0,,The file also contains st= andardize metadata Dialogue: 0,0:03:55.12,0:03:57.74,Default,,0,0,0,,and tool related configur= ation data Dialogue: 0,0:03:57.74,0:03:59.74,Default,,0,0,0,,which I'm not showing here Dialogue: 0,0:03:59.74,0:04:06.74,Default,,0,0,0,,A PEP 517* compatible bui= ld backend Dialogue: 0,0:04:06.74,0:04:10.56,Default,,0,0,0,,provides a standard funct= ion entrypoint Dialogue: 0,0:04:10.56,0:04:12.56,Default,,0,0,0,,called `build_wheel` Dialogue: 0,0:04:12.56,0:04:17.00,Default,,0,0,0,,in the module I just refe= renced here in the top Dialogue: 0,0:04:17.00,0:04:21.42,Default,,0,0,0,,and, if we call it, it wi= ll just do its magic Dialogue: 0,0:04:21.42,0:04:24.06,Default,,0,0,0,,and it will produce a whe= el file Dialogue: 0,0:04:24.06,0:04:28.30,Default,,0,0,0,,and its first argument it= 's the wheel directory Dialogue: 0,0:04:28.30,0:04:32.22,Default,,0,0,0,,that wheel is basically a= zip file Dialogue: 0,0:04:32.22,0:04:33.94,Default,,0,0,0,,with a predefined structu= re Dialogue: 0,0:04:33.94,0:04:37.30,Default,,0,0,0,,that we can extract into = our store Dialogue: 0,0:04:37.30,0:04:40.12,Default,,0,0,0,,and we are almost done Dialogue: 0,0:04:40.12,0:04:44.76,Default,,0,0,0,,and this is what Pip does= in the install phase here Dialogue: 0,0:04:50.36,0:04:54.48,Default,,0,0,0,,and that's basically the = entire build process Dialogue: 0,0:04:54.48,0:04:57.42,Default,,0,0,0,,as specified by PEP 517 Dialogue: 0,0:04:57.42,0:05:00.50,Default,,0,0,0,,there is no `setup.py` in= volved any more Dialogue: 0,0:05:00.50,0:05:01.50,Default,,0,0,0,,we don't have to call it Dialogue: 0,0:05:01.50,0:05:06.78,Default,,0,0,0,,we don't have to create i= t as a package provider Dialogue: 0,0:05:08.16,0:05:14.08,Default,,0,0,0,,so the reason why the err= or message I showed, showed up earlier Dialogue: 0,0:05:14.08,0:05:16.46,Default,,0,0,0,,will keep on poping up mo= re and more Dialogue: 0,0:05:16.52,0:05:19.34,Default,,0,0,0,,is pretty simple we are l= ate! Dialogue: 0,0:05:19.34,0:05:22.46,Default,,0,0,0,,we are really really late= , actually Dialogue: 0,0:05:22.46,0:05:28.72,Default,,0,0,0,,because PEP=C2=A0517 was = originally created in 2015 Dialogue: 0,0:05:28.72,0:05:34.14,Default,,0,0,0,,and that it gained provis= ional acceptance in 2017 Dialogue: 0,0:05:34.14,0:05:36.14,Default,,0,0,0,,and just last year, Dialogue: 0,0:05:36.14,0:05:42.74,Default,,0,0,0,,after being basically bei= ng the *de facto* successor of `setup.py` for some time Dialogue: 0,0:05:42.74,0:05:46.32,Default,,0,0,0,,it has been finalized and= fully accepted Dialogue: 0,0:05:46.32,0:05:53.18,Default,,0,0,0,,and more importantly, fli= t which you remember from the previous slide Dialogue: 0,0:05:53.18,0:05:57.26,Default,,0,0,0,,is also able to create so= urce distributions Dialogue: 0,0:05:57.26,0:05:59.54,Default,,0,0,0,,and upload them to PyPI Dialogue: 0,0:05:59.54,0:06:03.80,Default,,0,0,0,,Python's public package r= epository basically Dialogue: 0,0:06:03.80,0:06:08.08,Default,,0,0,0,,so far, it has been gener= ating a `setup.py` Dialogue: 0,0:06:08.08,0:06:10.88,Default,,0,0,0,,and does nobody really no= ticed Dialogue: 0,0:06:10.88,0:06:17.74,Default,,0,0,0,,but since version 3.5, wh= ich was released in November 2021 Dialogue: 0,0:06:17.74,0:06:21.88,Default,,0,0,0,,flit stop doing that by d= efault Dialogue: 0,0:06:21.88,0:06:26.98,Default,,0,0,0,,and thus we are seeing mo= re and more packages without `setup.py` Dialogue: 0,0:06:26.98,0:06:28.98,Default,,0,0,0,,in their source distribut= ions Dialogue: 0,0:06:28.98,0:06:33.78,Default,,0,0,0,,and so we are basically u= nable to build this projects right now Dialogue: 0,0:06:33.78,0:06:35.78,Default,,0,0,0,,or this Python modules Dialogue: 0,0:06:35.78,0:06:40.98,Default,,0,0,0,,a look at the Guix's repo= sitory in late January Dialogue: 0,0:06:40.98,0:06:45.26,Default,,0,0,0,,and back then only 11 pac= kages actually used Pip Dialogue: 0,0:06:45.26,0:06:47.84,Default,,0,0,0,,or PyPA build as we've se= en Dialogue: 0,0:06:47.84,0:06:51.80,Default,,0,0,0,,but I think our ecosystem= is quite old Dialogue: 0,0:06:51.80,0:06:56.68,Default,,0,0,0,,with about half the packa= ges not being the latest versions available upstream Dialogue: 0,0:06:56.68,0:06:58.68,Default,,0,0,0,,according to `guix refres= h` Dialogue: 0,0:06:58.68,0:07:05.56,Default,,0,0,0,,so it's possible that mor= e packages actually require support for this `pyproject.toml` Dialogue: 0,0:07:05.56,0:07:10.40,Default,,0,0,0,,and we simply have not up= dated them yet for whatever reason Dialogue: 0,0:07:10.40,0:07:14.82,Default,,0,0,0,,maybe because it's too di= fficult or nobody poped to do it yet Dialogue: 0,0:07:16.82,0:07:22.16,Default,,0,0,0,,they are also more issues= with our current Python build system Dialogue: 0,0:07:22.16,0:07:29.74,Default,,0,0,0,,for instance `setup.py` t= est ???? has been deprecated for quite some time Dialogue: 0,0:07:29.74,0:07:33.14,Default,,0,0,0,,since 2019 actually Dialogue: 0,0:07:33.14,0:07:39.12,Default,,0,0,0,,and the default Python bu= ild system's default check phase relies on that Dialogue: 0,0:07:39.12,0:07:42.72,Default,,0,0,0,,even though it doesn't wo= rk for a lot of packages right now Dialogue: 0,0:07:42.72,0:07:47.12,Default,,0,0,0,,and thus, almost every pa= ckage in our repository Dialogue: 0,0:07:47.12,0:07:52.12,Default,,0,0,0,,needs to replace that che= ck phase with a call to Pytest Dialogue: 0,0:07:52.12,0:07:56.40,Default,,0,0,0,,which is the *de facto* s= tandard right now for executing tests Dialogue: 0,0:07:56.40,0:08:03.02,Default,,0,0,0,,???? tox does something s= imilar but not really quite the same Dialogue: 0,0:08:03.02,0:08:08.68,Default,,0,0,0,,another long standing iss= ue is wrapping of Python path Dialogue: 0,0:08:08.68,0:08:10.68,Default,,0,0,0,,or Guix Python path now Dialogue: 0,0:08:10.68,0:08:13.52,Default,,0,0,0,,because it includes nativ= e inputs Dialogue: 0,0:08:13.52,0:08:16.94,Default,,0,0,0,,and thus propagates way t= oo many packages Dialogue: 0,0:08:16.94,0:08:22.30,Default,,0,0,0,,and I'm sure we can find = more issues with the current Python build system Dialogue: 0,0:08:22.30,0:08:34.40,Default,,0,0,0,,thus, for almost a year n= ow, I've been working on a modernized Python build system Dialogue: 0,0:08:34.40,0:08:37.00,Default,,0,0,0,,that addresses some of th= is issues Dialogue: 0,0:08:37.00,0:08:41.68,Default,,0,0,0,,unfortunately I haven't h= ad much time lately to improve it for other Dialogue: 0,0:08:41.68,0:08:43.58,Default,,0,0,0,,I hope to pick it up again Dialogue: 0,0:08:43.58,0:08:44.96,Default,,0,0,0,,and get it into shape Dialogue: 0,0:08:44.96,0:08:46.96,Default,,0,0,0,,and get it to master at s= ome point Dialogue: 0,0:08:46.96,0:08:48.96,Default,,0,0,0,,you can read about it here Dialogue: 0,0:08:48.96,0:08:54.48,Default,,0,0,0,,in Guix issue #46848 Dialogue: 0,0:08:54.48,0:09:00.88,Default,,0,0,0,,or you can just pull a br= anch called `wip-python-pep517` Dialogue: 0,0:09:00.88,0:09:03.56,Default,,0,0,0,,it's also build by the CI= by the way Dialogue: 0,0:09:03.56,0:09:07.70,Default,,0,0,0,,so you don't have to do t= he whole world rebuild by yourself Dialogue: 0,0:09:07.70,0:09:12.62,Default,,0,0,0,,and it's using exactly th= e method I described above Dialogue: 0,0:09:12.92,0:09:20.66,Default,,0,0,0,,the part written in Guile= ends up reading the project's `pyproject.toml` Dialogue: 0,0:09:20.66,0:09:28.58,Default,,0,0,0,,and then calls the build = system's entrypoint `build_wheel` as we can see here Dialogue: 0,0:09:28.58,0:09:34.84,Default,,0,0,0,,and if there's no `pyproj= ect.toml` it's also not a problem actually Dialogue: 0,0:09:34.84,0:09:40.10,Default,,0,0,0,,because setuptools, which= is still provided by default, Dialogue: 0,0:09:40.10,0:09:43.10,Default,,0,0,0,,and still included in the= build system by default, Dialogue: 0,0:09:43.10,0:09:49.26,Default,,0,0,0,,because setuptools provid= es a PEP 517 build backend Dialogue: 0,0:09:51.16,0:09:57.00,Default,,0,0,0,,which simply uses the exi= sting `setup.py` behind the curtains Dialogue: 0,0:09:57.00,0:10:02.68,Default,,0,0,0,,so we don't have to, like= provide extra logic for that Dialogue: 0,0:10:04.30,0:10:09.00,Default,,0,0,0,,they were actually other = options for this build phase Dialogue: 0,0:10:09.00,0:10:11.00,Default,,0,0,0,,in how to do it basically Dialogue: 0,0:10:12.14,0:10:15.36,Default,,0,0,0,,for example, we could als= o use the build module Dialogue: 0,0:10:15.36,0:10:16.98,Default,,0,0,0,,that we have seen above Dialogue: 0,0:10:16.98,0:10:19.96,Default,,0,0,0,,and some packages use cur= rently Dialogue: 0,0:10:19.96,0:10:23.50,Default,,0,0,0,,however that raises quest= ions about bootstraping, like Dialogue: 0,0:10:23.50,0:10:28.10,Default,,0,0,0,,how do you build the buil= d package if it essentially requires itself Dialogue: 0,0:10:28.10,0:10:31.70,Default,,0,0,0,,because the build module = is part of the Python build system Dialogue: 0,0:10:31.70,0:10:34.74,Default,,0,0,0,,and of of it's dependency Dialogue: 0,0:10:34.74,0:10:38.76,Default,,0,0,0,,tomli for example is one = of the dependency of build Dialogue: 0,0:10:38.76,0:10:42.10,Default,,0,0,0,,and what do we do with to= mli Dialogue: 0,0:10:42.10,0:10:44.90,Default,,0,0,0,,like how do we install it= without having build Dialogue: 0,0:10:44.90,0:10:49.26,Default,,0,0,0,,so this is tricky, and I = tried it in the beginning Dialogue: 0,0:10:49.26,0:10:51.02,Default,,0,0,0,,and it works Dialogue: 0,0:10:51.02,0:10:55.28,Default,,0,0,0,,but we need to do like so= me weird manual copying Dialogue: 0,0:10:55.28,0:10:58.46,Default,,0,0,0,,and that's not really pre= tty to be honest Dialogue: 0,0:10:58.46,0:11:00.78,Default,,0,0,0,,we could also use Pip Dialogue: 0,0:11:00.78,0:11:04.12,Default,,0,0,0,,which is usually distribu= ted with Python Dialogue: 0,0:11:04.12,0:11:09.16,Default,,0,0,0,,also our python package b= undles Pip and setuptools Dialogue: 0,0:11:09.16,0:11:12.36,Default,,0,0,0,,this has caused a few iss= ues in the past Dialogue: 0,0:11:12.36,0:11:14.70,Default,,0,0,0,,because we cannot update = either of them Dialogue: 0,0:11:14.70,0:11:17.78,Default,,0,0,0,,for provide alternative n= ewer versions Dialogue: 0,0:11:17.78,0:11:22.18,Default,,0,0,0,,without having a full wor= ld rebuild basically Dialogue: 0,0:11:22.18,0:11:25.40,Default,,0,0,0,,and also Pip being a mass= ive project Dialogue: 0,0:11:25.40,0:11:28.38,Default,,0,0,0,,it bundles quite a few of= it's dependencies Dialogue: 0,0:11:28.38,0:11:30.72,Default,,0,0,0,,and I feel "we" as a Guix= project Dialogue: 0,0:11:30.72,0:11:34.66,Default,,0,0,0,,should try to untangle th= at mess whenever we can Dialogue: 0,0:11:34.66,0:11:36.66,Default,,0,0,0,,and in this case, we actu= ally can Dialogue: 0,0:11:36.66,0:11:42.22,Default,,0,0,0,,so, in conclusion, I thin= k running this 4 lines of Python code Dialogue: 0,0:11:42.22,0:11:44.22,Default,,0,0,0,,that you can see here Dialogue: 0,0:11:45.88,0:11:48.82,Default,,0,0,0,,is a much simpler solution Dialogue: 0,0:11:48.82,0:11:51.98,Default,,0,0,0,,than trying to bootstrap = the build module, Dialogue: 0,0:11:51.98,0:12:00.24,Default,,0,0,0,,and tomli and all the mod= ules required to get a Python based builder basically Dialogue: 0,0:12:00.24,0:12:03.02,Default,,0,0,0,,we can just do it in 4 li= nes of code Dialogue: 0,0:12:06.52,0:12:09.76,Default,,0,0,0,,there are a few changes f= or packagers Dialogue: 0,0:12:09.76,0:12:13.34,Default,,0,0,0,,which aime to reduce the = need for custom phases Dialogue: 0,0:12:13.34,0:12:22.66,Default,,0,0,0,,first, it is possible to = override the build backend detected from the `pyproject.toml` Dialogue: 0,0:12:22.66,0:12:27.00,Default,,0,0,0,,especially early adopters= of PEP 517 Dialogue: 0,0:12:27.00,0:12:30.94,Default,,0,0,0,,use modules which have be= en renamed in the meantime Dialogue: 0,0:12:32.66,0:12:36.08,Default,,0,0,0,,I think both Poetry and F= lit did that at some point Dialogue: 0,0:12:36.08,0:12:40.62,Default,,0,0,0,,and they split their buil= d logic into a separate project Dialogue: 0,0:12:40.62,0:12:42.62,Default,,0,0,0,,and renamed the backend Dialogue: 0,0:12:42.62,0:12:48.72,Default,,0,0,0,,so, for some packages, it= 's necessary to override this detected build backend Dialogue: 0,0:12:48.72,0:12:52.60,Default,,0,0,0,,and you can do that simpl= y by specifying the `#:build-backend` option Dialogue: 0,0:12:52.60,0:12:56.30,Default,,0,0,0,,and also if the detection= is wrong at some point Dialogue: 0,0:12:56.30,0:13:00.16,Default,,0,0,0,,we can just override it w= ithout causing too much damage Dialogue: 0,0:13:01.10,0:13:05.68,Default,,0,0,0,,secondly, we have `#:conf= igure-flags` Dialogue: 0,0:13:05.98,0:13:10.46,Default,,0,0,0,,they have existed before,= but where applied only during install Dialogue: 0,0:13:10.46,0:13:12.14,Default,,0,0,0,,but not during the build = phase Dialogue: 0,0:13:12.14,0:13:15.30,Default,,0,0,0,,which, I believe, is a ??= ??, we'll see Dialogue: 0,0:13:15.30,0:13:20.06,Default,,0,0,0,,and this are kind of argu= ments to `setup.py` Dialogue: 0,0:13:20.14,0:13:23.64,Default,,0,0,0,,but their syntax depends = on the build backend unfortunately Dialogue: 0,0:13:23.64,0:13:27.80,Default,,0,0,0,,so sometimes you can pass= the arguments directly Dialogue: 0,0:13:27.80,0:13:31.46,Default,,0,0,0,,sometimes you need to pas= s them as an option to another option Dialogue: 0,0:13:31.46,0:13:33.32,Default,,0,0,0,,like we can see here Dialogue: 0,0:13:33.32,0:13:37.28,Default,,0,0,0,,I think that's the setupt= ools specific way to do it Dialogue: 0,0:13:37.28,0:13:39.54,Default,,0,0,0,,so you have an option Dialogue: 0,0:13:39.54,0:13:44.48,Default,,0,0,0,,and then you have to pass= the actual option as an argument to that option Dialogue: 0,0:13:44.48,0:13:50.60,Default,,0,0,0,,and, in theory, they coul= d also look completly different Dialogue: 0,0:13:50.60,0:13:53.98,Default,,0,0,0,,like with no relationship= to any command line flag Dialogue: 0,0:13:53.98,0:13:58.94,Default,,0,0,0,,it's just legacy code/stu= ff right now Dialogue: 0,0:13:58.94,0:14:00.50,Default,,0,0,0,,and it's not pretty Dialogue: 0,0:14:00.50,0:14:05.06,Default,,0,0,0,,and it's one of the thing= s that have been criticized about PEP=C2=A0517 Dialogue: 0,0:14:05.06,0:14:10.78,Default,,0,0,0,,this configure flags are = not really specified how they are supposed to look Dialogue: 0,0:14:10.78,0:14:16.18,Default,,0,0,0,,but this solution exists,= so we have to swim with the flow, kind of Dialogue: 0,0:14:17.00,0:14:20.96,Default,,0,0,0,,and, finally, they are `#= :test-flags` Dialogue: 0,0:14:20.96,0:14:28.48,Default,,0,0,0,,my proposal was to auto d= etect test binary based on the package's native inputs Dialogue: 0,0:14:28.48,0:14:32.92,Default,,0,0,0,,so if pytest is available= , it will just use pytest Dialogue: 0,0:14:32.92,0:14:35.74,Default,,0,0,0,,if nose is available, it = will use nose Dialogue: 0,0:14:35.74,0:14:37.16,Default,,0,0,0,,and so on Dialogue: 0,0:14:37.16,0:14:43.22,Default,,0,0,0,,now, test flags override = the default arguments passed to this test command Dialogue: 0,0:14:43.22,0:14:47.54,Default,,0,0,0,,and, hopefully, this can = reduce the number of custom phases required Dialogue: 0,0:14:47.54,0:14:49.70,Default,,0,0,0,,to exclude tests mostly Dialogue: 0,0:14:49.70,0:14:55.82,Default,,0,0,0,,in this case, we are simp= ly don't want to run the `test_legacy_callbacks` function Dialogue: 0,0:14:55.82,0:14:59.46,Default,,0,0,0,,and so we can just exclud= e it with the flags Dialogue: 0,0:15:01.38,0:15:05.72,Default,,0,0,0,,and the package will buil= d just fine, probably, hopefully Dialogue: 0,0:15:05.72,0:15:09.92,Default,,0,0,0,,so, where do we go from h= ere? Dialogue: 0,0:15:09.92,0:15:16.08,Default,,0,0,0,,the first step would be t= o finalize the work on the Python build system Dialogue: 0,0:15:16.08,0:15:23.16,Default,,0,0,0,,and get the branch ready = into a state where it can be merged Dialogue: 0,0:15:23.16,0:15:28.50,Default,,0,0,0,,unfortunately, right now,= quite a few packages fail to build there Dialogue: 0,0:15:28.50,0:15:34.04,Default,,0,0,0,,because their previously = disabled test suite fails Dialogue: 0,0:15:34.04,0:15:37.36,Default,,0,0,0,,not necessarily disabled,= but it was never ran Dialogue: 0,0:15:37.36,0:15:43.92,Default,,0,0,0,,because the test runner w= as buggy and the `setup.py test` command simply did nothing Dialogue: 0,0:15:43.92,0:15:46.24,Default,,0,0,0,,and didn't detect edge ca= ses Dialogue: 0,0:15:46.24,0:15:51.38,Default,,0,0,0,,we could also go a differ= ent route and add a new build system Dialogue: 0,0:15:51.38,0:15:53.22,Default,,0,0,0,,without touching the old = one Dialogue: 0,0:15:53.22,0:15:57.80,Default,,0,0,0,,and just slowly migrate e= very Python package we have over Dialogue: 0,0:15:57.80,0:15:59.90,Default,,0,0,0,,instead of doing the big = switch at once Dialogue: 0,0:16:00.02,0:16:03.30,Default,,0,0,0,,that was also one of the = suggestions that was certainly possible Dialogue: 0,0:16:03.30,0:16:05.76,Default,,0,0,0,,I have explored that it w= ould work Dialogue: 0,0:16:05.76,0:16:14.98,Default,,0,0,0,,as I mentioned earlier, c= urrently the native inputs end up in wrapped binaries Dialogue: 0,0:16:14.98,0:16:23.38,Default,,0,0,0,,and I think this is one t= hing we should address too while doing the whole world rebuild anyway Dialogue: 0,0:16:23.38,0:16:28.36,Default,,0,0,0,,but I haven't done any wo= rk on that yet, unfortunately Dialogue: 0,0:16:28.36,0:16:34.98,Default,,0,0,0,,and finally, the pypi imp= ort needs support to read `pyproject.toml` Dialogue: 0,0:16:34.98,0:16:38.74,Default,,0,0,0,,not so much for the runti= me dependencies, Dialogue: 0,0:16:38.74,0:16:43.88,Default,,0,0,0,,this is fine, we have ano= ther metadata file which contains the runtime dependencies Dialogue: 0,0:16:43.88,0:16:46.76,Default,,0,0,0,,but for build time and te= st dependencies Dialogue: 0,0:16:46.76,0:16:49.84,Default,,0,0,0,,because they are not incl= uded in the wheel file, Dialogue: 0,0:16:49.84,0:16:55.80,Default,,0,0,0,,there's a file called met= adata, and it doesn't have this data Dialogue: 0,0:16:55.80,0:17:03.78,Default,,0,0,0,,so right now, if you impo= rt a `pyproject.toml` based project Dialogue: 0,0:17:03.78,0:17:08.70,Default,,0,0,0,,you will lack the build d= ependencies basically Dialogue: 0,0:17:08.70,0:17:13.98,Default,,0,0,0,,for example flit will not= be in the native inpust right now Dialogue: 0,0:17:13.98,0:17:19.36,Default,,0,0,0,,and that's it for me for = now Dialogue: 0,0:17:19.36,0:17:24.60,Default,,0,0,0,,I hope to answer some of = your questions during the live Q and A Dialogue: 0,0:17:24.60,0:17:30.38,Default,,0,0,0,,and in the PDF version of= this slides, you'll find some clickable links bellow Dialogue: 0,0:17:30.38,0:17:36.34,Default,,0,0,0,,to different talks and to= the standards I have referenced Dialogue: 0,0:17:36.34,0:17:38.34,Default,,0,0,0,,so, I hope I'll see you s= oon! --===============5605966078630222987==--