Issue about building python on a foreign distro

Issue about building python on a foreign distro

[Wade Zhang]

Hi all,

Recently I installed Guix on a Debian buster host.  I want to install all build
dependencies by Guix.  Since I use pyenv to manage Python versions, I have to
install Python outside Guix.  Now I get stuck during the Python installation.

I install dependencies by the following command:

```
guix install glibc gcc-toolchain make pkg-config libffi zlib bzip2
readline openssl sqlite xz xmlsec libxml2
```

# With ccache

At first I install ccache to speed up the building:

```
guix install ccache
```

When I run `pyenv install 3.8.11`, I got the following error:

```
checking for getaddrinfo... yes
checking getaddrinfo bug... yes
Fatal: You must get working getaddrinfo() function.
       or you can specify "--disable-ipv6".
```

I can build Python with packages installed by apt, so I think it's not an ipv6
problem.  I also tried to `apt-get install nscd` but the building still failed.

# Without ccache

Then I uninstall ccache and run `pyenv install 3.8.11`, I got another error:

```
*** WARNING: renaming "_ctypes" since importing it failed:
build/lib.linux-x86_64-3.8/_ctypes.cpython-38-x86_64-linux-gnu.so:
undefined symbol: ffi_prep_cif

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_dbm                  _gdbm                 _lzma
_sqlite3              _tkinter              readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for
the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_uuid                 nis


Following modules built successfully but were removed because they
could not be imported:
_ctypes
```

It seems all dependencies installed by Guix are not found.

I'm new to Guix, I have searched these message but can't find the solution.

Could you please tell me the reason of these errors and how to fix them?

Best Regards,

Wade Zhang

Re: Issue about building python on a foreign distro

[zimoun]

Hi,

Welcome! :-)

On Sun, 22 Aug 2021 at 23:39, Wade Zhang <dochang@gmail.com> wrote:

> Recently I installed Guix on a Debian buster host.  I want to install all build
> dependencies by Guix.  Since I use pyenv to manage Python versions, I have to
> install Python outside Guix.  Now I get stuck during the Python installation.

[...]

> It seems all dependencies installed by Guix are not found.

Well, do you do

  source ~/.guix-profile/etc/profile

after installing all the dependencies (gcc-toolchain, bzip2, etc.)

> Could you please tell me the reason of these errors and how to fix them?

I do not use 'pyenv' and I had bad experience when mixing "package
manager".  To have other Python versions than the default ones, I
would create variant by using package transformation

  guix build python
--with-source=python=https://www.python.org/ftp/python/3.8.11/Python-3.8.11.tar.xz

but there is no guarantee it works for all the versions. Well, I would
start by a package transformation at the command-line and if it fails
then compile --without-tests or create a package variant and use
--load-path.


Hope that helps,
simon

Re: Issue about building python on a foreign distro

[Wade Zhang]

On Tue, Aug 24, 2021 at 12:41 AM zimoun <zimon.toutoune@gmail.com> wrote:
>
> Hi,
>
> Welcome! :-)

Thank you for your reply!

> > It seems all dependencies installed by Guix are not found.
>
> Well, do you do
>
>   source ~/.guix-profile/etc/profile
>
> after installing all the dependencies (gcc-toolchain, bzip2, etc.)

I'm sure I did.  Is there anything I missed?

```
doc@VM-8-2-debian:~$ env | grep ^PATH
PATH=/home/doc/.yarn/bin:/home/doc/.cargo/bin:/home/doc/bin:/home/doc/.anyenv/envs/pyenv/shims:/home/doc/.anyenv/envs/pyenv/bin:/home/doc/.anyenv/bin:/home/doc/.guix-profile/bin:/home/doc/.guix-profile/sbin:/home/doc/.config/guix/current/bin:/home/doc/.guix-profile/bin:/home/doc/.guix-profile/sbin:/home/doc/.config/guix/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
doc@VM-8-2-debian:~$ env | grep GUIX_PROFILE
GUIX_PROFILE=/home/doc/.guix-profile
_GUIX_PROFILE=/home/doc/.config/guix/current
doc@VM-8-2-debian:~$ command -v gcc
/home/doc/.guix-profile/bin/gcc
```

>
> > Could you please tell me the reason of these errors and how to fix them?
>
> I do not use 'pyenv' and I had bad experience when mixing "package
> manager".  To have other Python versions than the default ones, I
> would create variant by using package transformation

pyenv allows you to set python version for your every project by putting a
`.python-version` file, and you can easily override the version with an
environment variable.  All you need to do is just setting your `PATH`.  I think
it's very useful when you're working with multiple projects.

>
>   guix build python
> --with-source=python=https://www.python.org/ftp/python/3.8.11/Python-3.8.11.tar.xz
>
> but there is no guarantee it works for all the versions. Well, I would
> start by a package transformation at the command-line and if it fails
> then compile --without-tests or create a package variant and use
> --load-path.
>
>
> Hope that helps,
> simon

Re: Issue about building python on a foreign distro

[zimoun]

Hi,

On Tue, 24 Aug 2021 at 03:30, Wade Zhang <dochang@gmail.com> wrote:

>> > It seems all dependencies installed by Guix are not found.

I do not know.  I have tried without success:

--8<---------------cut here---------------start------------->8---
$ guix environment --pure python --ad-hoc bash curl nss-certs zlib
[env]$ export PYENV_ROOT="$HOME/.pyenv"
[env]$ export PATH="$PYENV_ROOT/bin:$PATH"
[env]$ eval "$(pyenv init --path)"
[env]$ pyenv install 3.8.2
--8<---------------cut here---------------end--------------->8---

and it fails about ’zlib’.

>> > Could you please tell me the reason of these errors and how to fix them?
>>
>> I do not use 'pyenv' and I had bad experience when mixing "package
>> manager".  To have other Python versions than the default ones, I
>> would create variant by using package transformation
>
> pyenv allows you to set python version for your every project by putting a
> `.python-version` file, and you can easily override the version with an
> environment variable.  All you need to do is just setting your `PATH`.  I think
> it's very useful when you're working with multiple projects.

Personally, I store the output of “guix describe -f channels” in say the
file channel.scm and I list all the packages the project require in say
the file manifest.scm.  Then I can exchange these 2 files are to produce
an environment (temporary profile) or a profile.  For instance:

  guix time-machine -C channels.scm -- environment -m manifest.scm

or

  guix time-machine -C channels.scm -- package -m manifest.scm -p profile

It becomes really easy to work with multiple projects. :-)  Note that
these 2 files are generally versioned with the whole project.  Even,
sometime I have several channels.scm files and manifest.scm files for
the same project in order to test.

The main drawback with the approach is the limitation to the Guix
packages.  For instance, two versions of CPython, only.  It is really
easy to use more packages from Conda for example.  I guess ’pyenv’
provides similar thing.  But you loose on other side. :-)

People use ’direnv’ with Guix to easily switch from context/project.
Another story. :-)


All the best,
simon

Re: Issue about building python on a foreign distro

[Wade Zhang]

On Wed, Aug 25, 2021 at 9:19 PM zimoun <zimon.toutoune@gmail.com> wrote:
> > pyenv allows you to set python version for your every project by putting a
> > `.python-version` file, and you can easily override the version with an
> > environment variable.  All you need to do is just setting your `PATH`.  I think
> > it's very useful when you're working with multiple projects.
>
> Personally, I store the output of “guix describe -f channels” in say the
> file channel.scm and I list all the packages the project require in say
> the file manifest.scm.  Then I can exchange these 2 files are to produce
> an environment (temporary profile) or a profile.  For instance:
>
>   guix time-machine -C channels.scm -- environment -m manifest.scm
>
> or
>
>   guix time-machine -C channels.scm -- package -m manifest.scm -p profile
>
> It becomes really easy to work with multiple projects. :-)  Note that
> these 2 files are generally versioned with the whole project.  Even,
> sometime I have several channels.scm files and manifest.scm files for
> the same project in order to test.
>

These days I've learned Guix commands and packaging guides.  I've seen the
power of Guix.

I found that Guix has another advantage that It can manage any other
dependencies, for example curl or sqlite, if the project requires them.

But it's still weird that Guix cannot be used with non-Guix packages...

Anyway thank you for your advice.  I'll try it.

> The main drawback with the approach is the limitation to the Guix
> packages.  For instance, two versions of CPython, only.  It is really
> easy to use more packages from Conda for example.  I guess ’pyenv’
> provides similar thing.  But you loose on other side. :-)

Yes, pyenv can install all versions of CPython and other implementations.  It
seems that Guix just removes old releases.  Although IMO I can use the feature
called transformation.

>
> People use ’direnv’ with Guix to easily switch from context/project.
> Another story. :-)
>
>
> All the best,
> simon

Best Regards,
Wade

Re: Issue about building python on a foreign distro

[Hartmut Goebel]

Am 22.08.21 um 13:27 schrieb Wade Zhang:
> Recently I installed Guix on a Debian buster host.  I want to install all build
> dependencies by Guix.  Since I use pyenv to manage Python versions, I have to
> install Python outside Guix.  Now I get stuck during the Python installation.

I never used pyenv, but Python's virtual environments a lot. Lately I 
moved on to "direnv" for most projects, as this matches my "directory 
based" working style. Maybe you want to give it a try?

A typical ".envrc" file for me looks like this (combining both a guix 
profile and a virtual env for developing the current package)

...8<-------
use_guix --load-path=../guix-channel \
      -v 1 --keep-failed \
      --ad-hoc glibc-utf8-locales python-wrapper python-pip 
python-virtualenv \
      python-pep-adapter sequoia python-pyyaml python-requests \
      python-setuptools python-setuptools-scm python-wheel python-babel \
      python-pytest@5.3.5 python-pytest-cov python-pytest-mp \
      python-requests-mock python-filelock python-pexpect

layout_python3 --system-site-packages
pip install -e .
path_add PYTHONPATH $PWD
...8<-------


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |