Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via <help-guix@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 6509 bytes --]

Hi again!

I accidently just replied to you, Simon, instead of making a "reply
all". I'm reposting the same now, sorry for the nuisance...

Thanks for your effort explaining email message ids m(_ _)m

> I am confused because, if I understand correctly, this tarball generated
> under ./dist is built using ’python3 -m build -s’, so from my
> understanding it is not the “normal Guix way”.  

OK, it seems I forgot to mention 1 thing - `python3 -m build -s` does
not really "build" a Python package. It builds a Python source tarball.
Like the ones that are pulled from PyPI as part of the Guix packaging
of many (most?) Python libraries available. The `python3 -m build`
command, without `-s`, would be used to build a Python wheel which I
suppose you thought I was doing.

> The point is to pack this definition…
> [...]
> …instead of this one.
>
>
> Could you give a try?  Something along the commands proposed by ’(’ in
> [1].  

Although I know it cannot help with my problem, for the reasons I wrote
to "(" in [1], I will do so for the sake of politeness.

So, I did run `guix shell -L. hydrilla`. First, I got a warning about

> ambiguous package specification `hydrilla'  

And a message indicating it chose the `hydrilla-dist-tarball`
definition. This is consistent with what I knew about package
resolution. So I now tried with
`guix shell -L. -e (@ (hydrilla) hydrilla)`. Also, I knew the build
would fail due to setuptools_scm being unable to find the `git`
command, so I temporarily added git to the native-inputs of `hydrilla`.

I got a failure in `sanity-check` phase. I saw that failure before -
this is what made me use `python3 -m build -s` in the first place, as I
described in [1]. The error was

> starting phase `sanity-check'
> validating 'hydrilla' /gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages
> ...checking requirements: OK
> ...trying to load module hydrilla: OK
> ...trying to load endpoint console_scripts haketilo: ERROR:
> Traceback (most recent call last):
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2458, in resolve
>     return functools.reduce(getattr, self.attrs, module)
> AttributeError: module 'hydrilla.mitmproxy_launcher.launch' has no attribute 'launch'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/gnu/store/35ix1m6m8a5s21j02ajhdyqxb2xkshfb-sanity-check.py", line 85, in <module>
>     ep.load()
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2450, in load
>     return self.resolve()
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2460, in resolve
>     raise ImportError(str(exc)) from exc
> ImportError: module 'hydrilla.mitmproxy_launcher.launch' has no attribute 'launch'  

That was followed by analogous errors for every entry point in the package.

I verified using `less` that
`/gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages/hydrilla/mitmproxy_launcher/launch.py`
is an empty file. Most other files in there are also empty but not all.
For example,
`/gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages/hydrilla/server/malcontent.py`
has proper contents.

As I said, this is the same problem I experienced before. To avoid any
ambiguity - using `hydrilla` recipe instead of `hydrilla-dist-tarball`
causes Guix to use the entirety of sources from current directory which
means
* `.git/` is included (it has to be for setuptools_scm to be able to
  cope with a git checkout that does not contain
  `src/hydrilla.egg-info/`)
* `src/hydrilla/_version.py` and `src/hydrilla.egg-info/` may also get
  included if they are present (i.e. if `python3 -m build -s` was
  already run at least once in git sources) but they are going to be
  ignored by setuptools_scm when Guix starts building the package
  because it sees `.git/`

Of course, the `hydrilla` package definition works properly when used
in an unpacked source tarball of my project (as opposed to git
chcekout). That's what I intended it for, after all.

Anyway, whether I use the `hydrilla` definition from my unpacked source
tarball or the `hydrilla-dist-tarball` definition from git checkout, it
all works well, namely
* guix environment/shell command builds my project properly and the
  `haketilo` command works inside the shell
* guix pack -RR builds a working pack that I can successfully use and
  that I also successfully tried out on a Debian Buster system

The problem that made me create this thread - that an end user fails to
use the pack on his system[2] and Python interpreter from inside the pack
behaves as if hydrilla from inside the pack was not on `PYTHONPATH` (nor
`GUIX_PYTHONPATH`, I guess) - remains a mystery.

The case of Guix putting empty files in a package when `.git/` is
included in the sources is indeed interesting. But right now it is not
really important - including git metadata in Guix input is not the
right thing to do when other options (such as using source tarball)
exist. I'd rather get back to the initial question - about possible
explanations why pack's Python interpreter might be using an invalid
Python library load path??

Disclaimer: in the experiments above I used a git checkout with 2
new commits[1] in it, hence the version of
`3.0b2.dev1+g9f26ebf.d20221013` instead of `3.0b1` as before. The
commits are not related to Guix packaging  - no need to look for the
cause of problems here :)

Best,
Wojtek

[1] https://yhetil.org/guix/20221013115135.2a82894d@koszkonutek-tmp.pl.eu.org/
[2] https://hydrillabugs.koszko.org/issues/130
[3] https://git.koszko.org/pydrilla/commit/?h=koszko&id=6c1f8221d17b1f4d7955d48a77fefeaf6e3030d7

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #20: saint Józef Sebastian Pelczar
Poznaj świętych krakowskich!  #20: święty Józef Sebastian Pelczar
https://pl.wikipedia.org/wiki/Józef_Sebastian_Pelczar
-- (sig_end)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]