From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id qLkHLY8VOWC0EgAA0tVLHw (envelope-from ) for ; Fri, 26 Feb 2021 15:36:47 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id CCrhKI8VOWChXwAAbx9fmQ (envelope-from ) for ; Fri, 26 Feb 2021 15:36:47 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2671CF4E4 for ; Fri, 26 Feb 2021 16:36:47 +0100 (CET) Received: from localhost ([::1]:49426 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lFfAc-0005om-9a for larch@yhetil.org; Fri, 26 Feb 2021 10:36:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34996) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lFfAT-0005od-Dg for guix-devel@gnu.org; Fri, 26 Feb 2021 10:36:37 -0500 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:44700) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lFfAR-0001Ao-9S; Fri, 26 Feb 2021 10:36:37 -0500 Received: by mail-qk1-x72f.google.com with SMTP id 204so9395582qke.11; Fri, 26 Feb 2021 07:36:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=xmCYLZwtZTRlDl+SuscsOmgKHAu5mul4mYNrNStQDnM=; b=YcNrChe1n4pgdAkH1ULZSk3o8NKsfFH38MFSfj1UtaRaFPXFjLkU++QZKa2tMNg4X2 6SKe6TrbiGQdTOd1ss0O0s/04DIaEYwflTDphtjaLDkxcefTCMp6Nq+to8Z4IouWEwGH ARzcfQ4UlyjqBgl0r0AwXrM6wdEGaTPWdpQMUy+jyoO04I5bHKJaWw7TXMHqCUG5Qzrp l2+KbWXZMQQmB/+iinB/YP+lgezLfICfEmSlVlJO2tomj9o4klwowbJzp6uXrnKDC9uu 9YiqvDrhFQnOd9e/i7FvExGwClVVWPvu7L0gnTl6rKpkR6gnHBWTFx0L55oIlTx86Npu jVxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=xmCYLZwtZTRlDl+SuscsOmgKHAu5mul4mYNrNStQDnM=; b=Wsa94Aj/BokIo4T9YyNGfrTSfyLR0JYHpUPjppTNIwW5FBhDgpmdQaZ6N+kulRb2cq UlDgOMCx1HlNXEH5SNQ7lVdzUNweqxbG/wli82u5uihQIjvLwd9X7lDGC7eS7BWA+vZv MU2HEviDEsINaMbRHUf0p32hdXiNBTe/+8cz3RVpX9OizDNRUfoCPzq9weKt4iRqLSXA 5dTA4Teb6mOGywyOSXyOoYIpZjMX2eBL6SgBGKtdyTKcNupnqDMYnKFvuPnv3G3rr8j2 wwWvpLlTdZ27qbTvOPC7ZpKoyQ2zfqdRIqlNawBD/N+jM9U+oEeKTdReNh9GqobvtecX QrEw== X-Gm-Message-State: AOAM530hyHTt0QGclB2YKS1wgHctzbYRW7fnRKPQFqGAJQSu3UgFxSg0 vMJR/QPyVa27gf8gLtVN6sSUZMRj9wnStA== X-Google-Smtp-Source: ABdhPJxjQEzWlLcC3RTHkujxj/XtDEC3XTE+UWEF/QlStGGyRJ3lIpt3+Zs+edxJvfgUewfwi9nwKQ== X-Received: by 2002:a37:4589:: with SMTP id s131mr3123877qka.269.1614353793088; Fri, 26 Feb 2021 07:36:33 -0800 (PST) Received: from hurd (dsl-10-134-178.b2b2c.ca. [72.10.134.178]) by smtp.gmail.com with ESMTPSA id p6sm6486950qkg.36.2021.02.26.07.36.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Feb 2021 07:36:32 -0800 (PST) From: Maxim Cournoyer To: Hartmut Goebel Subject: Re: 03/163: build/python: Add a new guix-pythonpath procedure. References: <20210125070022.22870.17321@vcs0.savannah.gnu.org> <20210125070114.03C0B20E1C@vcs0.savannah.gnu.org> <87czxs3d1i.fsf@gnu.org> <878s8g3bxa.fsf@gmail.com> <87lfcdm9jb.fsf@gnu.org> <87im7fyg5c.fsf@gmail.com> Date: Fri, 26 Feb 2021 10:36:31 -0500 In-Reply-To: (Hartmut Goebel's message of "Fri, 5 Feb 2021 11:26:20 +0100") Message-ID: <871rd296yo.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::72f; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x72f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1614353807; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=xmCYLZwtZTRlDl+SuscsOmgKHAu5mul4mYNrNStQDnM=; b=rmPCjqTJqjLNz/Jdq8b/E5fca6rGBoQGXTzmSP7oJJOqo3dIVwr6ZxJmS49t1ZRdQeXBZO 704RTL3tviKxwPEbQchvi5hSxv994B9Wl0WBnb2KqMhzLooU8bYaByp+XF2qwWyq0OyXkJ 6yiVxCgMG9rs83HH3Yvc1xz/TF4vqceICjZ38kHfXN+DXQq1lvmgHIrPZewCDLNC6WIILf 4eyYlz86DPnWpIULBETKMGUhg0WdsHJk6meTMms2ygreoGcj0qFPqkaI8BmYfkrVIdj9iC YrrPRD33re6ZVKCTjlO+W6YDYvZ8Up7F2kgo78CGwbY3yZZyY9LUEegKeirlMA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1614353807; a=rsa-sha256; cv=none; b=aQ/DYYJWxegTfs/FiplgN41OmA5perHQEAhV8YJnShhOPr88P4CJryrgdulImlb4QruPDl gRiQ9uklUVTxasdV2a8kFx3yC9REimbv4uOvBgIPadX7UdIw+IVyHzUjro2+x4n4m2H36W LsD5q8rZyr9FJXVt5DrvWOc5idAFcrx4bDi3fKTvvxIRXgYEgn9wzo0Fpu/Q6EX/eR5Plb NvUvtCnG5Q3V7VTknf7uDCrAsrCzKGjHpqZloNbLoWuXgQ+TlUN0UDlEL3ESRYi/X0nO6D 2a1TNy3bHDAOGdWYRScgLaCGPuTsBYyJGN+SqVcx6YtWGqoAjNhjkKE1JCsDmA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=YcNrChe1; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.27 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=YcNrChe1; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 2671CF4E4 X-Spam-Score: -1.27 X-Migadu-Scanner: scn0.migadu.com X-TUID: V+jXkiap//Ad Hi Hartmut, Sorry for the delay. Hartmut Goebel writes: > Hi Maxim, > > many thanks for picking up this issue. >> Indeed, I thought about the possibility to filter the GUIX_PYTHONPATH >> entries based on their version at runtime after I wrote my initial >> reply. It makes life easier. I've updated the >> cu/farewell-to-pythonpath branch with this new way of doing things. > > I had a look at the first changes (only) and have some remarks: > > 1) Did I understand this correctly: `sitecustomize.py` is filtering > GUIX_PYTHONPATH for all parts containing > "/lib/pythonX.Y/site-packages" (with X.Y being Major.Minor)? Yes. > 2) This does not remove duplicates and does not honor .pth files in > the respective directories - which might still be used. Thus > site.addsitedir() should be called for adding the paths. This also > takes care about duplicates. I confess I didn't pay attention to .pth files, which mostly seemed like legacy cruft to me; are they still used in the context of PEP 517 and modern Python packaging? The problem with calling site.addsitedir is that it simply appends to sys.path. We want to splice in the content of GUIX_PYTHONPATH at a controlled location. > 3) Empty part (=E2=80=A6::=E2=80=A6) are not handled. GUIX_PYTHONPATH is not intended for end users; users can and should still use PYTHONPATH, if they may. For that reason, I think it's preferable to keep it as lean as it can be, without burdening it with unnecessary checks. Empty sites also don't break anything, should they appear for any reason: $ PYTHONPATH=3D"::::$PYTHONPATH" python -c "print('hello\n')" hello > 4) Since PYTHONPATH is evaluated prior to importing sitecustomize, any > sitecustominze.py in the user's path will overwrite our file, thus > inhibiting our paths to be added. Not sure this is what we want in Guix. I asked guidance on the #python channel on freenode and was recommended to use sitecustomize.py for this purpose; reading the doc here seems to confirm our usage of it is as intended [0]: [...] After these path manipulations, an attempt is made to import a module named sitecustomize, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in the site-packages directory. The last sentence hints at that this is intended for the Python installation side of things rather than end users. I've never seen a Python package with a sitecustomize.py at its root; if I did, I would consider it bad practice. > 5) When implementing the logic into site.py, the code could be > simplified, Eg. You could patch a "getguixsitepackages" (based on > getsitepackages) and a "addguixsitepackages" (based on > addguixsitepackages) into site.py. Downside is that maybe we need > different patches for different Python versions. Benefit is simpler > code - no need to search the correct position in the list. I initially went that route, but ended up choosing sitecustomize.py as it is intended for this purpose and should be more robust in the face of changes to Python's site.py module. > 6) Please add some more comments to the code explaining the idea. I was under the impression the code was concise enough to forego with verbose explanations; I'd rather keep it this way. But as the implementer perhaps I'm not the right person to "see" this! I would be happy to consider a patch adding a few comments based on your fresh perspective, if you think it is worth it. > Some nitpicking: > >> python_root =3D os.path.realpath(sys.executable).split('/bin/')[0] > > There already is sys.prefix, which is also the base for the entry in > sys.path (see top of site.py Oh, true! >> major_minor =3D '{}.{}'.format(sys.version_info[0], sys.version_info[1]) > > major_minor =3D '{}.{}'.format(*sys.version_info) Neat trick; although the above is a bit more explicit. >>sys.path =3D sys.path[:index] + matching_sites + sys.path[index:] > > sys.path[index:index] =3D matching_sites Neat also! Thank you for pointing it out. > I suggest using os.path.join(), os.path.split(), os.pathsep, etc. to > be forward-compatible. Imagine we want to port Guix to another > platform with different separators. If Guix is ever ported to a system not using ':' as the path separator, it'd be a concern for the search path specification that emits GUIX_PYTHONPATH rather than here, no? Thanks for the comments and showing the neat little Python tricks above; I'd be happy to review a patch if you produce one with these changes and extra explanatory comments. Maxim [0] https://docs.python.org/3/library/site.html