From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 2HTaD0vG6GU3BQEA62LTzQ:P1 (envelope-from ) for ; Wed, 06 Mar 2024 20:38:51 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 2HTaD0vG6GU3BQEA62LTzQ (envelope-from ) for ; Wed, 06 Mar 2024 20:38:51 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=pVaAe2JM; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1709753931; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=+RabEAXAgHa9L1rhrN/a7d1jlWDvIMbKh6qsjrsaCWg=; b=otxzASx4GAtLE1nIJyQEdA+vV6iSM/idQGdOg0Tbk9YUrckvmUIjB6gEEnCiDM4jXK9IMu vrSUHR6fPHpcGC0Yq8ImurHVzJpRbhvlh5uEdraxB2KXCD+PTYAaPp1fT0Gwi5hEv/OOne QAbytpTShNJBX9jX6B63ZRCBIN3I6V4H803mK7xPPlBqdaQtDJEtuiHVBkqUV0oo3bnlaP kfyCVn3vYmsNVXqMf122sEFaKRA72n5x/Xbppnrjphr+yTzz86TdUpccNS+pz5/f2mUhTt il2AVxRz2CyI2LFMUT9syVnx6IQcCEn87zA2JlhvuzCccc1qta+uKvii/l3NlQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=pVaAe2JM; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1709753931; a=rsa-sha256; cv=none; b=NBfL3jDYMrK0xt13VZVL50ut0guNkHcXk+vLgyTkrDFs9uhx8f16HoTjsah0fZr8nWoZo/ /Aany4jv9RLuECa1ipy3GaF4kIosQ8Cs7xN6tvPhPsFzOJ6LPovrEqzsWXTZzKdC1vj6RX NN/j4FZTZcadikg0XdxXmp3SM7JwVYa3Ydmt0qRM40UraODBbF0lvpZupFZDhlpdMCzTUB LJSwVATcaV/aal4F+rbHir654mFpYEWBFulFuesoMhA+SFHKkp3NChB7AY+0YEG89lm66B 16fG+unn8wp2qHFFeM2jjkA5nTbORtJx6ZIZuy2B38/uWGDMVcwDPk5sjlWfJQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A000660131 for ; Wed, 6 Mar 2024 20:38:50 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rhx60-0001FY-Si; Wed, 06 Mar 2024 14:38:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5z-0001FB-3m for guix-patches@gnu.org; Wed, 06 Mar 2024 14:38:31 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rhx5y-0005WD-SG for guix-patches@gnu.org; Wed, 06 Mar 2024 14:38:30 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rhx6T-00084e-Pv for guix-patches@gnu.org; Wed, 06 Mar 2024 14:39:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69591] [PATCH 00/31] Unbundle and update python-pytorch Resent-From: David Elsing Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 06 Mar 2024 19:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 69591 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 69591@debbugs.gnu.org Cc: David Elsing X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.170975390430977 (code B ref -1); Wed, 06 Mar 2024 19:39:01 +0000 Received: (at submit) by debbugs.gnu.org; 6 Mar 2024 19:38:24 +0000 Received: from localhost ([127.0.0.1]:51488 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rhx5o-00083T-RI for submit@debbugs.gnu.org; Wed, 06 Mar 2024 14:38:24 -0500 Received: from lists.gnu.org ([209.51.188.17]:43452) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rhx5k-00083G-3S for submit@debbugs.gnu.org; Wed, 06 Mar 2024 14:38:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5E-00019s-RP for guix-patches@gnu.org; Wed, 06 Mar 2024 14:37:44 -0500 Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhx5C-0005PS-9w for guix-patches@gnu.org; Wed, 06 Mar 2024 14:37:44 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 5FBB8240028 for ; Wed, 6 Mar 2024 20:37:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1709753858; bh=UtszHbtRuqDAXRr8BuGVCJHzEvT/t0o/9sn52hTMd8g=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=pVaAe2JMVl7VEVJxt81N8EWSxypIiccje8ONOZDYFVrLbzMjQgRVsIr/GAWNL3BZY wDEKrWN0GMdU85dmIkwkGcuTwjseV1VgM3/ELvL1PlWIYE6xI2+YxZis6YeuM2PCIb k/bnxaNT1YmhTolHaPAe2i1gV3JWsjDm7Ozxs29OKUiAjFwoFfMiqmGUBjqhlnagry 6Sl5HsVhtZPpiiMFGtl7EweVjGHvpbgn1NXMZnamClGtdTiBR+pHj6BTgAuzehULtL MuZNxzso6bJklsMxSYwARmLQJouzvBr+Tp/14o++lujZKoPQWozYRByg5XEWPu5Myu NJ9StNUvfWN1w== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4TqjPs6b2dz9rxQ; Wed, 6 Mar 2024 20:37:37 +0100 (CET) From: David Elsing Date: Wed, 6 Mar 2024 19:37:23 +0000 Message-ID: <20240306193725.17463-1-david.elsing@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=185.67.36.65; envelope-from=david.elsing@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: 5.32 X-Spam-Score: 5.32 X-Migadu-Queue-Id: A000660131 X-TUID: /psNtN0MRoXo This patch series unbundles all dependencies of python-pytorch (updated to version 2.2.1) and removes all* autogenerated files, completing the work by Ludo' [1]. I also applied the unbundling to python-pytorch-for-r-torch (version 2.0.1) and made sure that r-torch still builds. Originally, I wanted to package PyTorch with ROCm, but decided that it would be better to first update the package and complete the unbundling. This was more work than I initially thought though, so this patch series is quite long. Caffe2 is heavily integrated in the PyTorch code, so unbundling it does not make sense. For QNNPACK, the situation is strange: There is an old version [2] and an internal fork in aten/src/ATen/native/quantized/cpu/qnnpack, which PyTorch both depends on (as well as XNNPACK). The internal fork does not depend on other parts of the code currently, so I unbundled it as well as a package variant of qnnpack. ideep [3] has a separate branch for PyTorch (the master branch does not seem maintained), which only contains header files without build instructions. Therefore, I made a simple package with the header files. miniz also requires a package variant for PyTorch. Several scripts for autogenerated files depend on the compiled PyTorch library. However, it was possible to patch the scripts with empty values and later regenerate the files with the initial PyTorch version. Therefore, I introduced the 'codegen2, 'build2 and 'install2 build phases running after the initial library is installed. In 'codegen2, the installed library is used to regenerate the files properly, which leads to a recompilation of only 12 files in the 'build2 phase. The core tests can be run as well, but I disabled them as they take a long time after an already long build. I also wanted to try and build python-pytorch on i686-linux, as it doesn't require emulation on an x86_64 CPU, but it depends on Rust through python-cryptography, for which there are no substitutes available currently. Cheers, David * with the exception of test/cpp/api/init_baseline.h and test/cpp/api/optim_baseline.h, which are just lists of previously calculated numbers; or any other generated file I might have missed. [1] https://hpc.guix.info/blog/2021/09/whats-in-a-package/ [2] https://github.com/pytorch/QNNPACK [3] https://github.com/intel/ideep/tree/pytorch David Elsing (31): gnu: asmjit: Update to commit 3ca5c18. gnu: Add python-typing-extensions-4.10. gnu: Add python-optree. gnu: Add python-pytest-flakefinder. gnu: Add python-pytest-shard. gnu: Add python-expecttest. gnu: Add python-pytest-rerunfailures-13. gnu: Add miniz. gnu: Add miniz-for-pytorch. gnu: Add libnop. gnu: Remove flatbuffers-next-shared. gnu: python-flatbuffers-next: Update to 23.5.26. gnu: pthreadpool: Update to commit 178e3e0. gnu: cpuinfo: Update to commit aa4b216. gnu: clog: Add "-DUSE_SYSTEM_LIBS=ON" configure flag. gnu: nnpack: Update to commit 70a77f4. gnu: oneapi-dnnl: Update to 3.3.5. gnu: Add tensorpipe. gnu: Add fbgemm. gnu: Add qnnpack. gnu: Add foxi. gnu: Add ideep-pytorch. gnu: xnnpack: Update to commit 51a9875. gnu: Remove xnnpack-for-torch2. gnu: Add qnnpack-pytorch. gnu: python-pytorch: Update to 2.2.1 and unbundle dependencies. gnu: python-torchvision: Update to 0.17.1. gnu: Add ideep-pytorch-for-r-torch. gnu: Add oneapi-dnnl-for-r-torch. gnu: Add qnnpack-pytorch-for-r-torch. gnu: python-pytorch-for-r-torch: Adjust to new python-pytorch. gnu/local.mk | 9 +- gnu/packages/compression.scm | 49 + gnu/packages/cpp.scm | 6 +- gnu/packages/machine-learning.scm | 874 +++- gnu/packages/parallel.scm | 30 +- .../patches/clog-fix-shared-build.patch | 39 +- .../patches/cpuinfo-system-libraries.patch | 50 - .../patches/fbgemm-use-system-libraries.patch | 62 + gnu/packages/patches/foxi-fix-build.patch | 55 + gnu/packages/patches/miniz-for-pytorch.patch | 47 + .../patches/python-optree-fix-32-bit.patch | 122 + ...ython-pytorch-1.9.0-system-libraries.patch | 139 - .../patches/python-pytorch-fix-codegen.patch | 167 + ...thon-pytorch-for-r-torch-fix-codegen.patch | 167 + ...pytorch-for-r-torch-system-libraries.patch | 390 ++ .../patches/python-pytorch-runpath.patch | 19 +- .../python-pytorch-system-libraries.patch | 324 +- .../python-pytorch-without-kineto.patch | 60 + .../python-pytorch2-system-libraries.patch | 156 - .../xnnpack-for-torch2-system-libraries.patch | 2660 ----------- .../patches/xnnpack-remove-broken-tests.patch | 335 ++ .../patches/xnnpack-system-libraries.patch | 4133 ++++++++++------- gnu/packages/python-build.scm | 12 + gnu/packages/python-check.scm | 88 + gnu/packages/python-xyz.scm | 29 + gnu/packages/serialization.scm | 53 +- 26 files changed, 5111 insertions(+), 4964 deletions(-) delete mode 100644 gnu/packages/patches/cpuinfo-system-libraries.patch create mode 100644 gnu/packages/patches/fbgemm-use-system-libraries.patch create mode 100644 gnu/packages/patches/foxi-fix-build.patch create mode 100644 gnu/packages/patches/miniz-for-pytorch.patch create mode 100644 gnu/packages/patches/python-optree-fix-32-bit.patch delete mode 100644 gnu/packages/patches/python-pytorch-1.9.0-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-without-kineto.patch delete mode 100644 gnu/packages/patches/python-pytorch2-system-libraries.patch delete mode 100644 gnu/packages/patches/xnnpack-for-torch2-system-libraries.patch create mode 100644 gnu/packages/patches/xnnpack-remove-broken-tests.patch -- 2.41.0