From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UGTQG9o8h2NblQAAbAwnHQ (envelope-from ) for ; Wed, 30 Nov 2022 12:22:02 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id wMnDG9o8h2NDCgEAauVa8A (envelope-from ) for ; Wed, 30 Nov 2022 12:22:02 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EC1F6134B9 for ; Wed, 30 Nov 2022 12:22:01 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p0L9X-0003QE-Tq; Wed, 30 Nov 2022 06:21:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p0L9W-0003PM-1c for help-guix@gnu.org; Wed, 30 Nov 2022 06:21:22 -0500 Received: from koszko.org ([93.95.227.159]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p0L9T-0003YO-CW for help-guix@gnu.org; Wed, 30 Nov 2022 06:21:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=koszko.org; s=mail; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Hz8fFlBbr/WsFeaXWz0gzfaEtJED2AL9wb97OH70tkc=; b=Jko40JyQb4oJUXqZJT4vzyjO/V pNjjPx1OtYmkTapkj1uOmVIY2SjxPVpkkLnbqvv3ZZJQGMZkoGcLmMTv17KKAa/t9Rdd/0ZAGCbwr ZPYNKdpwEOYsi4DK1p9/2h4xipQlQtN0X7Yp2ZSUsyaWQptLZ9uGTbdFofVX+NHo7QQwOnvR2SKu6 e6eUJUmhHV7wSSbi7n638TJe0FtS3qREpuZD2cj1UOljAiIll5NlQZ0w6CnA+Z+Lkr/JvhWJmm87Y gBPmvmA7OqyoE9/xKhoQWVcT4mh/TPZZdGKAzSrf5OYW9gAKJu++bNflsNt9rwroxWm32BsO2an5w c7BJMutcQLYDpiPZtepGPphDYrIiCzQlUWP9j+FLm5PExFB4RbAjhxkKXOeahIf5brodq6EhUtUdn 51TiLAlgPWf98t5JoW+XqHCCSDEuKzy1aR0as3riY27axA3yYHnCpheb1E/YU4B/r2R9hQFe1XIvf gLKsGcDvlWApg/8JGs1vXLUVCZVgWqhU81Ktrayw4y7e7/6x72/0brSJ5Cw7TQ2WtkrWnwO/IP+gU ejNp630svBCsIYiBwf4c6Tf0OjiRWfllQvTlmAxVkXLr+dh2th0mG1x/nl7HXpdA6cYLzjSHdc4t4 SF/wmzaKuVchEpSDzIz7RunmrouQIAHD3rMgo/HsY=; Received: from [77.252.47.212] (helo=koszkonutek-tmp.pl.eu.org) by koszko.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p0L9M-0002OJ-OK; Wed, 30 Nov 2022 12:21:12 +0100 Date: Wed, 30 Nov 2022 12:21:10 +0100 To: mprodrigues@posteo.net Cc: help-guix@gnu.org Subject: Re: Problem loading numpy and other on a venv in a guix system Message-ID: <20221130122110.708c7e20@koszkonutek-tmp.pl.eu.org> In-Reply-To: References: <3ee102246f0716dc9e4d1330062639ca@posteo.net> <20221129193335.1f263845@koszkonutek-tmp.pl.eu.org> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/nxjTl_90Vd3nM6k+HpaydKv"; protocol="application/pgp-signature"; micalg=pgp-sha256 Received-SPF: pass client-ip=93.95.227.159; envelope-from=koszko@koszko.org; helo=koszko.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Wojtek Kosior From: Wojtek Kosior via Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: help-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669807322; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Hz8fFlBbr/WsFeaXWz0gzfaEtJED2AL9wb97OH70tkc=; b=YTOCF8yWCB6Xosxk9lU0fK+8vwGszdhvEJKBDSAAjp3TOv3BUgNXOdngWfCdLHC5rFQTjg LFyb85mpthZofMsFImVIIypusW0djm+Y/QhoC0UAtBaF3bP2bu1aYasVCsY6BT570umYpQ RWL8eN39/hYPU1S2QxY5fqU4X0U60pPL2XtEh4+W/aNGwSfgqA2qAolYCkRFaPl2SkA6cA WIvgOWurofO2XQQzcWXo1WH96Ah04QFIPFF3nn3aOkJqLmPrfQqFlNm47BIRGvj8s4rwaw 4ElUNhBPu+Ry9YHtX6rN9dy+XEmerkrnoBdEfhKrUnMAlUNlCwUb9zWyVPGuMg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669807322; a=rsa-sha256; cv=none; b=bIYSSuKwu5TGrO6BmzOUYIty1Q8EgNhlS7WcRJw9wpa8QWUaAZz5yVi2TEbIBnjoV9WxBz y1sZnwbmhOcsojtDzs7FHg7oyvX67qzf8mTaVNOVXvCwiHnYuRpybQOy3Wut4wPuBqTyZ+ QESlXW5Q0qmt4y16Pw2Z0ji0mt5DZcdIWnXU9gMO1cLjHAz5CNpsWGrVVTM6Wfb2O34Xh6 OxhHahB80gEkpqVE8WWogBrR/5UlDYiyjdGsOrvF0HcMb138LhNfVmS0KkXVGln38VIcfl O0AMTJV8HdasKi11uKYBqrugqFI7/8c2Vr1ZZjR0x1/l0oaGBkqrWy3WgQFe1A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=koszko.org header.s=mail header.b=Jko40JyQ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.36 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=koszko.org header.s=mail header.b=Jko40JyQ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: EC1F6134B9 X-Spam-Score: -5.36 X-Migadu-Scanner: scn1.migadu.com X-TUID: i8gOI0J6NYz7 --Sig_/nxjTl_90Vd3nM6k+HpaydKv Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, > Hi Wojtek, >=20 > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue,=20 > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also=20 > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. >=20 > Best regards, > Mateus Rodrigues I see. I don't think I realize all the nuances of dynamic library loading myself. But I can try to explain what I know :) In C (also to some extent C++) world, libraries can be either linked statically into a program (i.e. included in its executable) or linked dynamically (i.e. so that they are found in the filesystem when program gets executed). There are pros and cons to both approaches. With dynamically-linked libraries (also called shared libraries), it is possible to write a program so that it doesn't load them immediately upon start. It is possible to instead have program's code call a C library function that loads a shared library *by name*, together with other shared libraries it depends on. This is what the Python interpreted process tries to do when told to load Numpy's C extension. Under a typical GNU/Linux system, shared libraries are searched for in some predefined locations (like `/lib/x86_64-linux-gnu/` which is where `libz.so.1` resides in my Debian OS). AFAIK, these locations are compiled into the C library and cannot be *changed*. However, they can be *supplemented* by paths in the `LD_LIBRARY_PATH` variable which are searched by the C library *before* the standard locations. Now, Guix does things differently. It does not keep its libraries in standard locations like `/lib`. It cannot - the assumption is that files needed by programs are kept under `/gnu/store`. So, AFAIU, the C library in Guix is *modified* not to assume any standard directories for shared libraries. Instead, all programs have the exact locations of their shared libraries specified directly in their executables. This is possible with the help of a (perhaps a bit obscure) feature called `RPATH`. So we already know that Guix' own packages make up for the lack of standard shared library directories. How about binaries like those from PyPI that were not compiled for Guix? They don't have the RPATH set and they expect libc to be able to resolve some of their dependencies. In particular, Python C extensions devs used to be rather safe to assume that zlib (a dependency of CPython) would be present and resolvable in the user's operating system and need not be bundled with the extension. If we think about it, an analogous problem appears for many languages, not just C. And it seems Guix solves it slightly differently there. For example, Python library locations are specified in Guix-prepared variable `GUIX_PYTHONPATH`. I believe that works more reliably than C's `RPATH` (although I had some issues with loading of Python libs, too). Back to your problem - symlinks to shared libraries are also placed in the `lib/` subdirectory of a Guix profile. For example, after doing guix install zlib I got a `libz.so.1` inside `~/.guix-profile/lib`. Perhaps this will prove helpful to you? :) Wojtek -- (sig_start) website: https://koszko.org/koszko.html PGP: https://koszko.org/key.gpg fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A Meet Krak=C3=B3w saints! #39: blessed Narcyz Turchan Poznaj =C5=9Bwi=C4=99tych krakowskich! #39: b=C5=82ogos=C5=82awiony Narcyz= Turchan https://pl.wikipedia.org/wiki/Narcyz_Turchan -- (sig_end) On Wed, 30 Nov 2022 09:42:34 +0000 mprodrigues@posteo.net wrote: > Hi Wojtek, >=20 > thanks for your help, indeed setting LD_LIBRARY_PATH solves the issue,=20 > but it is less than ideal as I have to first > check exactly the PATH to set, and if I update guix this will also=20 > change I believe, still it is not so bad. I'd still like > to understand why this happens in the first place. >=20 > Best regards, > Mateus Rodrigues >=20 > On 29.11.2022 19:33, Wojtek Kosior wrote: > > Hi Mateus! > >=20 > > Indeed, when using virtualenv's numpy, this > >=20 > > python3 -c 'import numpy' > >=20 > > fails with the error you posted. I seem to have been able to make numpy > > happy by instead running Python like > >=20 > >=20 > > LD_LIBRARY_PATH=3D/gnu/store/8qv5kb2fgm4c3bf70zcg9l6hkf3qzpw9-zlib-1.2.= 11/lib/ > > python3 -c 'import numpy' > >=20 > > If you use the `--container` and `--emulate-fhs` options to > > `guix shell`, it is also possible to instead do > >=20 > > LD_LIBRARY_PATH=3D/usr/lib/ python3 -c 'import numpy' > >=20 > > The `libz` guix package still needs to be included among the packages > > passed to `guix shell` in this last case. > >=20 > > Also, given you're using virtualenv, you might find the `--network` > > option to `guix shell` useful :) > >=20 > > I hope there isn't any dumb mistake in my solution... Good luck :) > >=20 > > Wojtek > >=20 > > -- (sig_start) > > website: https://koszko.org/koszko.html > > PGP: https://koszko.org/key.gpg > > fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A > >=20 > > Meet Krak=C3=B3w saints! #27: blessed Maria Klemensa Staszews= ka > > Poznaj =C5=9Bwi=C4=99tych krakowskich! #27: b=C5=82ogos=C5=82awiona Ma= ria Klemensa=20 > > Staszewska > > https://pl.wikipedia.org/wiki/Maria_Klemensa_Staszewska > > -- (sig_end) > >=20 > >=20 > > On Tue, 29 Nov 2022 16:42:14 +0000 > > mprodrigues@posteo.net wrote: > > =20 > >> Hello, > >>=20 > >> I've been using a guix shell with a couple of python packages and it > >> works fine, > >> however I'd also like to keep the option of using a virtualenv if > >> possible, the > >> thing is I'm having problem running modules like numpy, that depend on > >> non > >> python dependencies. The error I get when I try to load numpy in a=20 > >> venv > >> is: > >>=20 > >> Importing the numpy C-extensions failed. This error can happen for=20 > >> many > >> reasons, > >> often due to issues with your setup or how NumPy was installed. > >>=20 > >> and in the end: > >>=20 > >> Original error was: libz.so.1: cannot open shared object file: No such > >> file or > >> directory > >>=20 > >> I have tried installing zlib directly, also tried to add my default > >> profile libs > >> path to the venv PATH, but to no avail until now. How can I solve=20 > >> this? > >> Or > >> maybe someone can point me in the direction that I have to take to > >> properly > >> debug this. > >>=20 > >> Best regards, > >> Mateus Rodrigues > >> =20 --Sig_/nxjTl_90Vd3nM6k+HpaydKv Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQTpcnBg48VjfIpPS0JLxSIcWnn9GgUCY4c8pgAKCRBLxSIcWnn9 Gh0RAQCYkAO9uwsTKQS1zp/MoYHWlvNOf9T34XYFdujmxkm9bgEA2tPRgetEUynP PIVKh7oQOtBRoVa53m/K1QAbRLHYhA8= =x4u0 -----END PGP SIGNATURE----- --Sig_/nxjTl_90Vd3nM6k+HpaydKv--