program prepared with `guix pack` unusable by end users

program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

Hello,

I'm developing a program, named Haketilo, in Python. I wanted to make it
available to users by the means of a tarball made with `guix pack`
(which seems like a marvelous feature, btw!). It worked well on my PC
but a user reported an error when running it[1]. The console_scripts
entry point of the program cannot be found by importlib.

User's entire report is behind the link[1] - I'll just quote the
relevant traceback of `./haketilo --version`

> Traceback (most recent call last):
>   File "/gnu/store/ziiryffcgph9jjcldz4rv2x5v6y0kxqh-hydrilla-3.0b1/bin/.haketilo-real", line 33, in <module>
>     sys.exit(load_entry_point('hydrilla==3.0b1', 'console_scripts', 'haketilo')())
>   File "/gnu/store/ziiryffcgph9jjcldz4rv2x5v6y0kxqh-hydrilla-3.0b1/bin/.haketilo-real", line 25, in importlib_load_entry_point
>     return next(matches).load()
> StopIteration

For an unknown reason, the user reports it works with sudo. Does anyone
have an idea what could be the cause?

I made the pack by running

> guix environment -L . -e '(@ (hydrilla) hydrilla)' -- python3 -m build -s
> guix pack -L . -RR \
>     -S /hydrilla=bin/hydrilla \
>     -S /hydrilla-builder=bin/hydrilla-builder \
>     -S /hydrilla-server=bin/hydrilla-server \
>     -S /haketilo=bin/haketilo \
>     -e '(@ (hydrilla) hydrilla-dist-tarball)'

in the git repo of my project[2] checked out to tag `v3.0-beta1`. The
Guix version used is 1.3.0-26.fd00ac7.

Any advice is appreciated :)

Best,
Wojtek

[1] https://hydrillabugs.koszko.org/issues/130
[2] https://git.koszko.org/pydrilla


-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #48: saint Szymon z Lipnicy
Poznaj świętych krakowskich!  #48: święty Szymon z Lipnicy
https://pl.wikipedia.org/wiki/Szymon_z_Lipnicy
-- (sig_end)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[(]

Heya,

On Thu Oct 13, 2022 at 8:17 AM BST, Wojtek Kosior via wrote:
> > guix environment -L . -e '(@ (hydrilla) hydrilla)' -- python3 -m build -s
> > guix pack -L . -RR \
> >     -S /hydrilla=bin/hydrilla \
> >     -S /hydrilla-builder=bin/hydrilla-builder \
> >     -S /hydrilla-server=bin/hydrilla-server \
> >     -S /haketilo=bin/haketilo \
> >     -e '(@ (hydrilla) hydrilla-dist-tarball)'

I don't think this is an optimal way to use ``guix pack''. Surely this pack
won't contain the Python modules that the script needs to import...?

Have you tried

  guix  pack -RL. hydrilla

? Note that you don't actually need to use ``-e'', as the UI finds packages
by scanning all the modules in the load path for public variables containing
<package> objects.

Also, ``guix environment'' has been deprecated for a while now; consider using
``guix shell'':

  guix environment -L. hydrilla
  # becomes
  guix shell -DL. hydrilla

(``-D'' means "use the dependencies of the next listed package", as ``--ad-hoc''
is the default behaviour of ``guix shell''.)

    -- (

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 6032 bytes --]

Thanks for response, Paren

> Heya,
> 
> On Thu Oct 13, 2022 at 8:17 AM BST, Wojtek Kosior via wrote:
> > > guix environment -L . -e '(@ (hydrilla) hydrilla)' -- python3 -m build -s
> > > guix pack -L . -RR \
> > >     -S /hydrilla=bin/hydrilla \
> > >     -S /hydrilla-builder=bin/hydrilla-builder \
> > >     -S /hydrilla-server=bin/hydrilla-server \
> > >     -S /haketilo=bin/haketilo \
> > >     -e '(@ (hydrilla) hydrilla-dist-tarball)'  
> 
> I don't think this is an optimal way to use ``guix pack''. Surely this pack
> won't contain the Python modules that the script needs to import...?

Actually, the pack *does* contain all the necessary modules. Here I made
`hydrilla` resolve to a package that uses the entire contents of
project directory as its `(source)`. `hydrilla-dist-tarball` then
inherits from that package and causes a tarball under `./dist/` to
instead be used as `(source)`. All the `(propagated-inputs)` are there,
unmodified, in the child package.

Any other ideas what could be causing problems?

If you're curious, below I explain how I arrived at the commands I am
using. Note that this is almost certainly in no way related to how
console_script entries are found.

So, in my project I initially added a guix.scm[1] that I could
successfully use with `guix shell -Df guix.scm`. In guix.scm I
initially specified the source of "hydrilla" package as

> (source (local-file %source-dir
>                     #:recursive? #t
>                     #:select? (git-predicate %source-dir)))

with %source-dir defined as

> (define %source-dir (dirname (current-filename)))

This turned out not to work correctly for building the package because
the setuptools_scm python build plugin I am using relies on git
metadata to decide the version of the package and what files should
belong to it. Metadata from `.git/` is of course excluded with
`git-predicate`. Also, an attempt to get away without using the
`#:select?` keyword argument at all caused even weirder problems with
some .py files being included in Guix package as empty files...

Anyway, I figured out the best solution is to first generate a source
tarball with `python3 -m build -s`. Such tarball gets placed under
`./dist/` and already contains all the metadata prepared by
setuptools_scm. Hence, the alternative definition of package source[2]

> (source (local-file
>          (string-append %source-dir "/dist/" filename)))

with `filename` bound to an appropriate string...

This does work. But the tarball from beneath `./dist/` is what I
distribute to users. Why should someone who downloads that tarball need
to use the `python -m build -s`? There's no need. So I ended up making
2 variants of the package definition. One for release tarball users,
using `(local-file %source-dir #:recursive? #t)`. And one for git
checkout users.

Now, I was not aware of a good way to use 2 different definitions with
a simple guix.scm and `-f` option. Hence, I made it into a module and
renamed that to hydrilla.scm. Under normal circumstances I could
probably get away without using the `-e` option to guix. But here I
have 2 packages with the same name "hydrilla". To be able to choose
which one I want to use, I needed to refer to them by the names under
which they are exported from the module. Hence, the `-e` option.

Seeing how much confusion all this causes, I think I will stop caring
about release tarball users running an unnecessary
`python -m build -s`...

> Also, ``guix environment'' has been deprecated for a while now; consider using
> ``guix shell'':
> 
>   guix environment -L. hydrilla
>   # becomes
>   guix shell -DL. hydrilla
> 
> (``-D'' means "use the dependencies of the next listed package", as ``--ad-hoc''
> is the default behaviour of ``guix shell''.)
> 
>     -- (

I experienced guix shell failing to pick up changes I was making to
guix.scm. It seemed to be incorrectly using a cached version of that
file. Well, perhaps this behavior shall no longer occur when I use the
`-e` flag? Idk, I discovered that flag after swithing to
`guix environment`. Being now reminded about the deprecation (thanks!),
I think I'll give `guix shell` another chance, though :)

Best,
Wojtek

[1] https://git.koszko.org/pydrilla/tree/guix.scm?id=d42379c189c33dac732a1a1341395a9f5683260b
[2] https://git.koszko.org/pydrilla/tree/hydrilla.scm?h=v3.0-beta1

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #2: blessed Aniela Salawa
Poznaj świętych krakowskich!  #2: błogosławiona Aniela Salawa
https://pl.wikipedia.org/wiki/Aniela_Salawa
-- (sig_end)


On Thu, 13 Oct 2022 09:26:32 +0100
"(" <paren@disroot.org> wrote:

> Heya,
> 
> On Thu Oct 13, 2022 at 8:17 AM BST, Wojtek Kosior via wrote:
> > > guix environment -L . -e '(@ (hydrilla) hydrilla)' -- python3 -m build -s
> > > guix pack -L . -RR \
> > >     -S /hydrilla=bin/hydrilla \
> > >     -S /hydrilla-builder=bin/hydrilla-builder \
> > >     -S /hydrilla-server=bin/hydrilla-server \
> > >     -S /haketilo=bin/haketilo \
> > >     -e '(@ (hydrilla) hydrilla-dist-tarball)'  
> 
> I don't think this is an optimal way to use ``guix pack''. Surely this pack
> won't contain the Python modules that the script needs to import...?
> 
> Have you tried
> 
>   guix  pack -RL. hydrilla
> 
> ? Note that you don't actually need to use ``-e'', as the UI finds packages
> by scanning all the modules in the load path for public variables containing
> <package> objects.
> 
> Also, ``guix environment'' has been deprecated for a while now; consider using
> ``guix shell'':
> 
>   guix environment -L. hydrilla
>   # becomes
>   guix shell -DL. hydrilla
> 
> (``-D'' means "use the dependencies of the next listed package", as ``--ad-hoc''
> is the default behaviour of ``guix shell''.)
> 
>     -- (



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[zimoun]

Hi,

Is it related to the message [1] you are answering?

1: <https://yhetil.org/guix/CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework/>

If no, please start a new thread when it is a new topic. :-)



On jeu., 13 oct. 2022 at 09:17, Wojtek Kosior via <help-guix@gnu.org> wrote:

> I made the pack by running
>
>> guix environment -L . -e '(@ (hydrilla) hydrilla)' -- python3 -m build -s
>> guix pack -L . -RR \
>>     -S /hydrilla=bin/hydrilla \
>>     -S /hydrilla-builder=bin/hydrilla-builder \
>>     -S /hydrilla-server=bin/hydrilla-server \
>>     -S /haketilo=bin/haketilo \
>>     -e '(@ (hydrilla) hydrilla-dist-tarball)'

Why do you pack ’hydrilla-dist-tarball’ instead of just ’hydrilla’.

Guix should take care of everything; not necessary when packing a Python
bundle as you are doing.


Cheers,
simon

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1.1: Type: text/plain, Size: 2604 bytes --]

Hi Simon, thanks for your response

> Is it related to the message [1] you are answering?
> 
> 1: <https://yhetil.org/guix/CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework/>
> 
> If no, please start a new thread when it is a new topic. :-)

It took me a couple of minutes to understand what you're talking about. Indeed, instead of copy-pasting the <help-guix@gnu.org> address I clicked "Reply" on some random email from Guix mailing list and changed the subject. For years I've been certain that messages are categorized into threads by their subject. Now, as you wrote this, I assume there must be some thread meta-data that is invisibly sent by our user agents when we use "Reply" or "Reply all".

Who would have thought? In the past, when I saw email software nicely present thread emails as a tree of responses, I was thinking "wow, this program must be using some really successful heuristic for determining what is the response to what". Lol

Doing as you requested. Although this is suboptimal as well since now
the topic is split between 2 threads :/

> Why do you pack ’hydrilla-dist-tarball’ instead of just ’hydrilla’.
> 
> Guix should take care of everything; not necessary when packing a
> Python bundle as you are doing.

It's ok, `hydrilla-dist-tarball` refers to a package that is built in
the package in the normal Guix way. The name just indicates that
tarball generated under `./dist/` is used instead of the project file
tree. If you're curious, I explain the crazy details in my previous
email response to "(".

It sometimes happens that to keep my message concise I write what seems
to be the most important and attach links to other stuff that the
reader can look at to understand everything. Here that "stuff" was the
project repo which contains hydrilla.scm with both package definitions.

And most of the time I learn that I failed to be clear enough and that
people don't have time to read my links. Well, I'm sorry for the
confusion. Although I'm starting to lose hope that I will learn to
communicate with ppl online without so many misunderstandings :/

The hydrilla.scm that I was loading with `-L .` is now attached. Is
there anything wrong in it?

Best,
Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #4: blessed Bronisław Markiewicz
Poznaj świętych krakowskich!  #4: błogosławiony Bronisław Markiewicz
https://pl.wikipedia.org/wiki/Bronisław_Markiewicz
-- (sig_end)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: hydrilla.scm --]
[-- Type: text/x-scheme, Size: 11972 bytes --]

;; SPDX-License-Identifier: CC0-1.0

;; Copyright (C) 2022 Wojtek Kosior <koszko@koszko.org>
;;
;; Available under the terms of Creative Commons Zero v1.0 Universal.

(define-module (hydrilla))

(use-modules
 (ice-9 rdelim)
 (ice-9 regex))

(define %source-dir (dirname (current-filename)))

;; The PKG-INFO file is generated when running `python3 -m build -s` or similar.
;; It is also automatically included in the generated tarballs.
(define %hydrilla-version
  (if (access? "src/hydrilla.egg-info/PKG-INFO" R_OK)
      (let* ((port (open-input-file "src/hydrilla.egg-info/PKG-INFO"))
             (process-line
              (lambda (self-ref)
                (let ((match-result
                       (string-match "^Version: (.*)" (read-line port))))
                  (if match-result (match:substring match-result 1)
                      (self-ref self-ref))))))
        (process-line process-line))
      "unknown"))

(use-modules
 (guix packages)
 (guix download)
 (guix git-download)
 (guix build-system python)
 (guix gexp)
 ((guix licenses) #:prefix license:)
 (gnu packages python-build)
 (gnu packages python-xyz)
 (gnu packages python-crypto)
 (gnu packages compression)
 (gnu packages python-compression)
 (gnu packages xdisorg)
 (gnu packages serialization)
 (gnu packages protobuf)
 (gnu packages python-web)
 (gnu packages check)
 (gnu packages sphinx)
 (gnu packages python-check)
 (gnu packages license)
 (gnu packages gnupg))

(define-public python-kaitaistruct
  (package
    (name "python-kaitaistruct")
    (version "0.10")
    (source
     (origin
       (method url-fetch)
       (uri (pypi-uri "kaitaistruct" version))
       (sha256
        (base32 "0ap5ka51gnc2mc4s1kqqsi6nb6zqv8wsrg17ryxazmkkj7idwi50"))))
    (build-system python-build-system)
    (home-page "https://kaitai.io")
    (native-inputs (list python-wheel))
    (synopsis
     "Declarative parser generator for binary data: runtime library for Python")
    (description
     "Kaitai Struct is a declarative language used for describing various binary
data structures, laid out in files or in memory - i.e. binary file formats,
network stream packet formats, etc.")
    (license license:expat)))

(define-public python-parver
  (package
    (name "python-parver")
    (version "0.3.1")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "parver" version))
        (sha256
          (base32 "1lyzqp8bz0n2kzabzl7k7g7cn90rlnrxjzva2p62gsfc7djy00n9"))))
    (build-system python-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'relax-requirements
           (lambda _
             (substitute* "setup.py"
                          (("arpeggio[^']*") "arpeggio"))))
         (replace 'check
           (lambda* (#:key tests? #:allow-other-keys)
             (when tests?
               (invoke "pytest")))))))
    (propagated-inputs (list python-arpeggio python-attrs python-six))
    (native-inputs
      (list python-hypothesis
            python-pretend
            python-pytest))
    (home-page "https://github.com/RazerM/parver")
    (synopsis "Parse and manipulate version numbers")
    (description "Parver facilitates parsing and manipulation of
@url{https://www.python.org/dev/peps/pep-0440/,PEP 440} version numbers.")
    (license license:expat)))

(define-public python-pyopenssl-for-haketilo
  (let ((base python-pyopenssl))
    (package
      (inherit base)
      (version "22.0.0")
      (source
       (origin
         (inherit (package-source base))
         (uri (pypi-uri "pyOpenSSL" version))
         (sha256
          (base32
           "1gzihw09sqi71lwx97c69hab7w4rbnl6hhfrl6za3i5a4la1n2v6"))))
      (propagated-inputs
       (modify-inputs (package-propagated-inputs base)
         (replace "python-cryptography" python-cryptography-next))))))

(define-public python-urllib3-for-haketilo
  (let ((base python-urllib3))
    (package
      (inherit base)
      (propagated-inputs
       (modify-inputs (package-propagated-inputs base)
         (replace "python-cryptography" python-cryptography-next)
         (replace "python-pyopenssl" python-pyopenssl-for-haketilo))))))

(define-public python-requests-for-haketilo
  (let ((base python-requests))
    (package
      (inherit base)
      (propagated-inputs
       (modify-inputs (package-propagated-inputs base)
         (replace "python-urllib3" python-urllib3-for-haketilo))))))

(define-public python-werkzeug-for-haketilo
  (let ((base python-werkzeug))
    (package
      (inherit base)
      (propagated-inputs
       (modify-inputs (package-propagated-inputs base)
         (replace "python-requests" python-requests-for-haketilo))))))

(define-public python-flask-for-haketilo
  (let ((base python-flask))
    (package
      (inherit base)
      (propagated-inputs
       (modify-inputs (package-propagated-inputs base)
         (replace "python-werkzeug" python-werkzeug-for-haketilo))))))

(define-public mitmproxy
  (package
    (name "mitmproxy")
    (version "8.1.1")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/mitmproxy/mitmproxy")
             (commit (string-append "v" version))))
       (sha256
        (base32 "0kpzk8ci02vyjg9nqnpnadmgyaxxrpdydgfnm2xmxf1s4rzdcvwx"))
       (snippet
        '(begin
           ;; The player contains some minified JS. It would be possible to find
           ;; player sources elsewhere on the internet but there's no point in
           ;; doing do since we're not building the docs anyway.
           (delete-file "docs/src/assets/asciinema-player.js")
           #t))))
    (build-system python-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'relax-requirements
           (lambda _
             (substitute* "setup.py"
               (("kaitaistruct>=0\\.7[^\"]*") "kaitaistruct")
               ;; The ">=2.8" req was there because older ldap3 lacked a crucial
               ;; ">=0.4.8" req for its dep, pyasn. It's not an issue for Guix
               ;; which ships with pyasn 4.8 anyway.
               (("ldap3>=2\\.8[^\"]*") "ldap3")
               (("protobuf>=3\\.14,<5") "protobuf")
               (("sortedcontainers>=2\\.3[^\"]*") "sortedcontainers")
               (("wsproto>=1\\.0[^\"]*") "wsproto")
               (("pytest-timeout[^\"]*<2[^\"]*") "pytest-timeout")
               (("pytest-asyncio[^\"]*<0.14[^\"]*") "pytest-asyncio"))
             (substitute* "test/mitmproxy/proxy/layers/http/test_http.py"
               (("isinstance\\(x, HTTPFlow\\)")
                "issubclass(type(x), HTTPFlow)"))))
         (replace 'check
           (lambda* (#:key tests? #:allow-other-keys)
             (when tests?
               (setenv "HOME" "/tmp")
               (invoke "pytest" "--timeout" "60")))))))
    (propagated-inputs
     (list python-asgiref
           python-blinker
           python-brotli
           python-cryptography-next
           python-flask-for-haketilo
           python-h11
           python-h2
           python-hyperframe
           python-kaitaistruct
           python-ldap3
           python-msgpack
           python-passlib
           python-protobuf
           python-pyopenssl-for-haketilo
           python-pyparsing
           python-pyperclip
           python-ruamel.yaml
           python-sortedcontainers
           python-tornado-6
           python-urwid
           python-wsproto
           python-publicsuffix2
           python-zstandard))
    (native-inputs
     (list python-parver
           python-pytest
           python-pytest-asyncio
           python-pytest-timeout))
    (home-page "https://mitmproxy.org/")
    (synopsis "A free interactive HTTPS proxy")
    (description
     "An interactive TLS-capable intercepting HTTP proxy for penetration testers
and software developers.  It can be used to intercept, inspect, modify and
replay web traffic such as HTTP/1, HTTP/2, WebSockets, or any other
SSL/TLS-protected protocols.")
    (license license:expat)))

(define-public python-immutables-for-haketilo
  (let ((base python-immutables))
    (package
      (inherit base)
      (version "0.19")
      (source
       (origin
         ;; Old version tarballs seem to be getting deleted from PyPI each time
         ;; a new version comes out.
         (method git-fetch)
         (uri (git-reference
               (url "https://github.com/MagicStack/immutables")
               (commit (string-append "v" version))))
         (sha256
          (base32
           "1awjylp4pl0jknwgnabg7kkj8f8883fkf99spsdrw1pj1acajvy9"))))
      (arguments
       `(#:phases
         (modify-phases %standard-phases
           (add-after 'unpack 'fix-expected-mypy-types
             (lambda _
               (substitute* "tests/test-data/check-immu.test"
                 (("builtins.str") "builtins.str*"))))
           (replace 'check
             (lambda* (#:key tests? #:allow-other-keys)
               (when tests?
                 (invoke "pytest")))))))
      (native-inputs
       (list python-pytest python-mypy)))))

(define-public python-types-requests
  (package
    (name "python-types-requests")
    (version "2.26.0")
    (source (origin
              (method url-fetch)
              (uri (pypi-uri "types-requests" version))
              (sha256
               (base32
                "10sq8jarr642vhw53k6zbf3hn2b8xfyrckwfngml4fj19g1whpnz"))))
    (build-system python-build-system)
    (home-page "https://github.com/python/typeshed")
    (synopsis "Typing stubs for requests")
    (description
     "This package provides a collection of library stubs for Python, with
static types.")
    (license license:asl2.0)))

;; Use this variant when building from a downloaded release tarball.
(define-public hydrilla
  (package
    (name "hydrilla")
    (version %hydrilla-version)
    (source (local-file %source-dir #:recursive? #t))
    (build-system python-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (replace 'check
           (lambda* (#:key tests? #:allow-other-keys)
             (when tests?
               (invoke "pytest")))))))
    (propagated-inputs
     (list mitmproxy
           python-beautifulsoup4
           python-click
           python-flask-for-haketilo
           python-gnupg
           python-html5lib
           python-immutables-for-haketilo
           python-itsdangerous
           python-jsonschema
           reuse))
    (native-inputs
      (list python-setuptools-scm
            python-babel
            python-pytest
            python-pypa-build
            python-mypy
            python-types-requests))
    (home-page "https://hydrillabugs.koszko.org/projects/haketilo/wiki")
    (synopsis "Block JavaScript and add custom logic to web pages")
    (description "Haketilo HTTP proxy facilitates viewing of websites while
having their original JavaScript replaced by user-provided scripts. Haketilo
combines the functionalities of content blocker and user script manager. It can
be used with its script repository, Hydrilla.")
    (license (list license:agpl3+ license:gpl3+ license:cc0))))

;; Use this variant when building from a tarball generated under dist/. This
;; can be used to build from a git checkout after running `python3 -m build -s`
;; or similar.
(define-public hydrilla-dist-tarball
  (let ((base hydrilla)
        (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
    (package
      (inherit base)
      (source (local-file
               (string-append %source-dir "/dist/" filename))))))

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[(]

Heya,

On Thu Oct 13, 2022 at 5:20 PM BST, Wojtek Kosior via wrote:
> It took me a couple of minutes to understand what you're talking about. Indeed, instead of copy-pasting the <help-guix@gnu.org> address I clicked "Reply" on some random email from Guix mailing list and changed the subject. For years I've been certain that messages are categorized into threads by their subject. Now, as you wrote this, I assume there must be some thread meta-data that is invisibly sent by our user agents when we use "Reply" or "Reply all".

Yes, the In-Reply-To: header :)

    -- (

Re: program prepared with `guix pack` unusable by end users

[zimoun]

Hi, Wojtek,
On jeu., 13 oct. 2022 at 18:20, Wojtek Kosior via <help-guix@gnu.org> wrote:

> It took me a couple of minutes to understand what you're talking
> about. Indeed, instead of copy-pasting the <help-guix@gnu.org> address
> I clicked "Reply" on some random email from Guix mailing list and
> changed the subject. For years I've been certain that messages are
> categorized into threads by their subject. Now, as you wrote this, I
> assume there must be some thread meta-data that is invisibly sent by
> our user agents when we use "Reply" or "Reply all".
>
> Who would have thought? In the past, when I saw email software nicely
> present thread emails as a tree of responses, I was thinking "wow,
> this program must be using some really successful heuristic for
> determining what is the response to what". Lol

The header of an email contains many information, for example the one
you replied in the other thread, 

--8<---------------cut here---------------start------------->8---
[...]
Date: Thu, 13 Oct 2022 07:33:06 +0100
Message-Id: <CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework>
Subject: Re: Greetd autologin?
From: "(" <paren@disroot.org>
To: "kiasoc5" <kiasoc5@disroot.org>, <help-guix@gnu.org>
References: <ab7e4e012324af3f1fc796874216837e@disroot.org>
In-Reply-To: <ab7e4e012324af3f1fc796874216837e@disroot.org>
[...]
--8<---------------cut here---------------end--------------->8---

and your message in this other thread,

--8<---------------cut here---------------start------------->8---

[...]

Received: from [77.252.47.255] (helo=koszkonutek-tmp.pl.eu.org)
 by koszko.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <koszko@koszko.org>) id 1oisT6-0007ax-Ue
 for help-guix@gnu.org; Thu, 13 Oct 2022 09:17:25 +0200
Date: Thu, 13 Oct 2022 09:17:22 +0200
To: <help-guix@gnu.org>
Subject: program prepared with `guix pack` unusable by end users
Message-ID: <20221013091722.59d9ecc0@koszkonutek-tmp.pl.eu.org>
In-Reply-To: <CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework>
References: <ab7e4e012324af3f1fc796874216837e@disroot.org>
 <CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework>
X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu)

[...]

List-Id: <help-guix.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/help-guix>,
 <mailto:help-guix-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/help-guix>
List-Post: <mailto:help-guix@gnu.org>
List-Help: <mailto:help-guix-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/help-guix>,
 <mailto:help-guix-request@gnu.org?subject=subscribe>
Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org
Sender: "Help-Guix" <help-guix-bounces+larch=yhetil.org@gnu.org>
Reply-to:  Wojtek Kosior <koszko@koszko.org>
From:  Wojtek Kosior via <help-guix@gnu.org>

[...]
--8<---------------cut here---------------end--------------->8---

and such headers are hidden by most of mail clients.


Quickly said, the thread is built using the fields Message-ID and
In-Reply-To.  If you give a look at the header of the message you
answered, it contains the Message-ID field:

    CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework

and your message contains the field In-Reply-To

    CNKL38E5T0RV.1VYM8R2V0O1QM@guix-framework

and that chain builds the thread, somehow.


> Doing as you requested. Although this is suboptimal as well since now
> the topic is split between 2 threads :/

Communication is never optimal. ;-)


>> Why do you pack ’hydrilla-dist-tarball’ instead of just ’hydrilla’.
>> 
>> Guix should take care of everything; not necessary when packing a
>> Python bundle as you are doing.
>
> It's ok, `hydrilla-dist-tarball` refers to a package that is built in
> the package in the normal Guix way. The name just indicates that
> tarball generated under `./dist/` is used instead of the project file
> tree. If you're curious, I explain the crazy details in my previous
> email response to "(".

I am confused because, if I understand correctly, this tarball generated
under ./dist is built using ’python3 -m build -s’, so from my
understanding it is not the “normal Guix way”.


> It sometimes happens that to keep my message concise I write what seems
> to be the most important and attach links to other stuff that the
> reader can look at to understand everything. Here that "stuff" was the
> project repo which contains hydrilla.scm with both package definitions.
>
> And most of the time I learn that I failed to be clear enough and that
> people don't have time to read my links. Well, I'm sorry for the
> confusion. Although I'm starting to lose hope that I will learn to
> communicate with ppl online without so many misunderstandings :/

Communication, especially when we are not native English, is not
straightforward. :-)


> The hydrilla.scm that I was loading with `-L .` is now attached. Is
> there anything wrong in it?

Thanks.  Note that I have already given a look before answering you. ;-)


> ;; Use this variant when building from a downloaded release tarball.
> (define-public hydrilla
>   (package
>     (name "hydrilla")
>     (version %hydrilla-version)
>     (source (local-file %source-dir #:recursive? #t))
>     (build-system python-build-system)
>     (arguments
>      `(#:phases
>        (modify-phases %standard-phases
>          (replace 'check
>            (lambda* (#:key tests? #:allow-other-keys)
>              (when tests?
>                (invoke "pytest")))))))
>     (propagated-inputs
>      (list mitmproxy
>            python-beautifulsoup4
>            python-click
>            python-flask-for-haketilo
>            python-gnupg
>            python-html5lib
>            python-immutables-for-haketilo
>            python-itsdangerous
>            python-jsonschema
>            reuse))
>     (native-inputs
>       (list python-setuptools-scm
>             python-babel
>             python-pytest
>             python-pypa-build
>             python-mypy
>             python-types-requests))
>     (home-page "https://hydrillabugs.koszko.org/projects/haketilo/wiki")
>     (synopsis "Block JavaScript and add custom logic to web pages")
>     (description "Haketilo HTTP proxy facilitates viewing of websites while
> having their original JavaScript replaced by user-provided scripts. Haketilo
> combines the functionalities of content blocker and user script manager. It can
> be used with its script repository, Hydrilla.")
>     (license (list license:agpl3+ license:gpl3+ license:cc0))))

The point is to pack this definition…

> ;; Use this variant when building from a tarball generated under dist/. This
> ;; can be used to build from a git checkout after running `python3 -m build -s`
> ;; or similar.
> (define-public hydrilla-dist-tarball
>   (let ((base hydrilla)
>         (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
>     (package
>       (inherit base)
>       (source (local-file
>                (string-append %source-dir "/dist/" filename))))))

…instead of this one.


Could you give a try?  Something along the commands proposed by ’(’ in
[1].

1: <https://yhetil.org/guix/CNKNI3CYETFG.2QP3RDK7DF7PD@guix-framework>


Cheers,
simon

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 6509 bytes --]

Hi again!

I accidently just replied to you, Simon, instead of making a "reply
all". I'm reposting the same now, sorry for the nuisance...

Thanks for your effort explaining email message ids m(_ _)m

> I am confused because, if I understand correctly, this tarball generated
> under ./dist is built using ’python3 -m build -s’, so from my
> understanding it is not the “normal Guix way”.  

OK, it seems I forgot to mention 1 thing - `python3 -m build -s` does
not really "build" a Python package. It builds a Python source tarball.
Like the ones that are pulled from PyPI as part of the Guix packaging
of many (most?) Python libraries available. The `python3 -m build`
command, without `-s`, would be used to build a Python wheel which I
suppose you thought I was doing.

> The point is to pack this definition…
> [...]
> …instead of this one.
>
>
> Could you give a try?  Something along the commands proposed by ’(’ in
> [1].  

Although I know it cannot help with my problem, for the reasons I wrote
to "(" in [1], I will do so for the sake of politeness.

So, I did run `guix shell -L. hydrilla`. First, I got a warning about

> ambiguous package specification `hydrilla'  

And a message indicating it chose the `hydrilla-dist-tarball`
definition. This is consistent with what I knew about package
resolution. So I now tried with
`guix shell -L. -e (@ (hydrilla) hydrilla)`. Also, I knew the build
would fail due to setuptools_scm being unable to find the `git`
command, so I temporarily added git to the native-inputs of `hydrilla`.

I got a failure in `sanity-check` phase. I saw that failure before -
this is what made me use `python3 -m build -s` in the first place, as I
described in [1]. The error was

> starting phase `sanity-check'
> validating 'hydrilla' /gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages
> ...checking requirements: OK
> ...trying to load module hydrilla: OK
> ...trying to load endpoint console_scripts haketilo: ERROR:
> Traceback (most recent call last):
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2458, in resolve
>     return functools.reduce(getattr, self.attrs, module)
> AttributeError: module 'hydrilla.mitmproxy_launcher.launch' has no attribute 'launch'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/gnu/store/35ix1m6m8a5s21j02ajhdyqxb2xkshfb-sanity-check.py", line 85, in <module>
>     ep.load()
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2450, in load
>     return self.resolve()
>   File "/gnu/store/b6j1qw1a5rkbfvcy7lc9fm95abbzpa4x-python-3.9.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2460, in resolve
>     raise ImportError(str(exc)) from exc
> ImportError: module 'hydrilla.mitmproxy_launcher.launch' has no attribute 'launch'  

That was followed by analogous errors for every entry point in the package.

I verified using `less` that
`/gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages/hydrilla/mitmproxy_launcher/launch.py`
is an empty file. Most other files in there are also empty but not all.
For example,
`/gnu/store/fj5ijdxsw6nz23ymxf397kd7d5h3pbrj-hydrilla-3.0b2.dev1+g9f26ebf.d20221013/lib/python3.9/site-packages/hydrilla/server/malcontent.py`
has proper contents.

As I said, this is the same problem I experienced before. To avoid any
ambiguity - using `hydrilla` recipe instead of `hydrilla-dist-tarball`
causes Guix to use the entirety of sources from current directory which
means
* `.git/` is included (it has to be for setuptools_scm to be able to
  cope with a git checkout that does not contain
  `src/hydrilla.egg-info/`)
* `src/hydrilla/_version.py` and `src/hydrilla.egg-info/` may also get
  included if they are present (i.e. if `python3 -m build -s` was
  already run at least once in git sources) but they are going to be
  ignored by setuptools_scm when Guix starts building the package
  because it sees `.git/`

Of course, the `hydrilla` package definition works properly when used
in an unpacked source tarball of my project (as opposed to git
chcekout). That's what I intended it for, after all.

Anyway, whether I use the `hydrilla` definition from my unpacked source
tarball or the `hydrilla-dist-tarball` definition from git checkout, it
all works well, namely
* guix environment/shell command builds my project properly and the
  `haketilo` command works inside the shell
* guix pack -RR builds a working pack that I can successfully use and
  that I also successfully tried out on a Debian Buster system

The problem that made me create this thread - that an end user fails to
use the pack on his system[2] and Python interpreter from inside the pack
behaves as if hydrilla from inside the pack was not on `PYTHONPATH` (nor
`GUIX_PYTHONPATH`, I guess) - remains a mystery.

The case of Guix putting empty files in a package when `.git/` is
included in the sources is indeed interesting. But right now it is not
really important - including git metadata in Guix input is not the
right thing to do when other options (such as using source tarball)
exist. I'd rather get back to the initial question - about possible
explanations why pack's Python interpreter might be using an invalid
Python library load path??

Disclaimer: in the experiments above I used a git checkout with 2
new commits[1] in it, hence the version of
`3.0b2.dev1+g9f26ebf.d20221013` instead of `3.0b1` as before. The
commits are not related to Guix packaging  - no need to look for the
cause of problems here :)

Best,
Wojtek

[1] https://yhetil.org/guix/20221013115135.2a82894d@koszkonutek-tmp.pl.eu.org/
[2] https://hydrillabugs.koszko.org/issues/130
[3] https://git.koszko.org/pydrilla/commit/?h=koszko&id=6c1f8221d17b1f4d7955d48a77fefeaf6e3030d7

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #20: saint Józef Sebastian Pelczar
Poznaj świętych krakowskich!  #20: święty Józef Sebastian Pelczar
https://pl.wikipedia.org/wiki/Józef_Sebastian_Pelczar
-- (sig_end)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[zimoun]

Hi,

On ven., 14 oct. 2022 at 11:09, Wojtek Kosior via <help-guix@gnu.org> wrote:

> I accidently just replied to you, Simon, instead of making a "reply
> all". I'm reposting the same now, sorry for the nuisance...

Do not worry. :-)

> So, I did run `guix shell -L. hydrilla`. First, I got a warning about
>
>> ambiguous package specification `hydrilla'

That’s expected because:

--8<---------------cut here---------------start------------->8---
(define-public hydrilla
  (package
    (name "hydrilla")

[...]

(define-public hydrilla-dist-tarball
  (let ((base hydrilla)
        (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
    (package
      (inherit base)
      (source (local-file
--8<---------------cut here---------------end--------------->8---

It means that the package ’hydrilla’ has the name “hydrilla” and because
the package ’hydrilla-dist-tarball’ inherits then it also has the name
“hydrilla”.  Idem for the version field.

Therefore,

    guix shell -L . hydrilla

refers to 2 possible packages.  All Guix CLI commands use “name“ and
“version” for looking up to packages and not the symbols.


Well, then I tried to reproduce your failure in order to understand what
is going wrong but I hit this:

--8<---------------cut here---------------start------------->8---
$ guix build -L . python-pyopenssl@22.0.0
The following derivations will be built:
  /gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv
  /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv
building /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv...
pyOpenSSL-22.0.0/

[...]

pyOpenSSL-22.0.0/tox.ini
patching file src/OpenSSL/SSL.py
Hunk #1 FAILED at 1421.
Hunk #2 FAILED at 2449.
2 out of 2 hunks FAILED -- saving rejects to file src/OpenSSL/SSL.py.rej
patching file tests/test_ssl.py
Hunk #1 FAILED at 1928.
1 out of 1 hunk FAILED -- saving rejects to file tests/test_ssl.py.rej
source is at 'pyOpenSSL-22.0.0'
applying '/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch'...
Backtrace:

[...]

ERROR:
  1. &invoke-error:
      program: "/gnu/store/z39hnrwds1dgcbpfgj8dnv2cngjb2xbl-patch-2.7.6/bin/patch"
      arguments: ("--force" "--no-backup-if-mismatch" "-p1" "--input" "/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch")

[...]

builder for `/gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv' failed with exit code 1
build of /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv failed
View build log at '/var/log/guix/drvs/rf/awn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv.gz'.
cannot build derivation `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv' failed
--8<---------------cut here---------------end--------------->8---

That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
is that the ’origin’ also inherits

       (origin
         (inherit (package-source base))
         
and thus

       (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))

does not make sense anymore because this patch is against 21.0.0 and
your variant is about 22.0.0.


Well, further investigations about what could be wrong for “guix pack”
needs a way to reproduce. :-)  Could you fix or point a working example
in your Git repository?


Thanks,
simon

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 6716 bytes --]

Hello again after a break,

> Well, then I tried to reproduce your failure in order to understand what
> is going wrong but I hit this:
>
> [...]
> 
> That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
> ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
> is that the ’origin’ also inherits
> 
>        (origin
>          (inherit (package-source base))
>          
> and thus
> 
>        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
> 
> does not make sense anymore because this patch is against 21.0.0 and
> your variant is about 22.0.0.

Thank you for your time spent on trying this out.

The error is surely a result of us using different Guix versions. In the
version I had the pyOpenSSL version was 20.0.1 and no patch was being
used yet[1].

Another user also had problems building from source under a different
Guix version. I knew this would happen but I didn't have a ready
solution before. I now came up with a Makefile that organizes most
common development tasks in my project into rules that call `guix
time-machine` under the hood[2]. This should be a general solution for
this kind of problems.

> Well, further investigations about what could be wrong for “guix pack”
> needs a way to reproduce. :-)  Could you fix or point a working
> example in your Git repository?

As for troubleshooting the creation of Hydrilla 3.0-beta1 pack, one can
get the same enviroment I was using by pulling Guix from commit
fd00ac7e27c8a6d4b4d4719b4f8797bdce204623. Although most list users here
probably know how to achieve this, I'm putting this here for convenience

> guix time-machine --commit=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623 -- <REST-OF-THE-USUAL-GUIX-COMMAND>

Anyway, the main problem could probably be also troubleshooted by
experimenting with the pack I made available[3] as it is also the one
that is known to have caused problems.

Thank you again,
Wojtek

[1] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-crypto.scm?id=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623#n650
[2] https://git.koszko.org/pydrilla/commit/?h=koszko&id=ed2e14dd6ced4f64b6ac707c5998b38a78e6d7ad
[3] https://hydrillabugs.koszko.org/projects/haketilo/wiki/Releases#Files

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #13: blessed Jan Beyzym
Poznaj świętych krakowskich!  #13: błogosławiony Jan Beyzym
https://pl.wikipedia.org/wiki/Jan_Beyzym
-- (sig_end)


On Fri, 14 Oct 2022 13:00:18 +0200
zimoun <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> On ven., 14 oct. 2022 at 11:09, Wojtek Kosior via <help-guix@gnu.org> wrote:
> 
> > I accidently just replied to you, Simon, instead of making a "reply
> > all". I'm reposting the same now, sorry for the nuisance...  
> 
> Do not worry. :-)
> 
> > So, I did run `guix shell -L. hydrilla`. First, I got a warning about
> >  
> >> ambiguous package specification `hydrilla'  
> 
> That’s expected because:
> 
> --8<---------------cut here---------------start------------->8---
> (define-public hydrilla
>   (package
>     (name "hydrilla")
> 
> [...]
> 
> (define-public hydrilla-dist-tarball
>   (let ((base hydrilla)
>         (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
>     (package
>       (inherit base)
>       (source (local-file
> --8<---------------cut here---------------end--------------->8---
> 
> It means that the package ’hydrilla’ has the name “hydrilla” and because
> the package ’hydrilla-dist-tarball’ inherits then it also has the name
> “hydrilla”.  Idem for the version field.
> 
> Therefore,
> 
>     guix shell -L . hydrilla
> 
> refers to 2 possible packages.  All Guix CLI commands use “name“ and
> “version” for looking up to packages and not the symbols.
> 
> 
> Well, then I tried to reproduce your failure in order to understand what
> is going wrong but I hit this:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build -L . python-pyopenssl@22.0.0
> The following derivations will be built:
>   /gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv
>   /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv
> building /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv...
> pyOpenSSL-22.0.0/
> 
> [...]
> 
> pyOpenSSL-22.0.0/tox.ini
> patching file src/OpenSSL/SSL.py
> Hunk #1 FAILED at 1421.
> Hunk #2 FAILED at 2449.
> 2 out of 2 hunks FAILED -- saving rejects to file src/OpenSSL/SSL.py.rej
> patching file tests/test_ssl.py
> Hunk #1 FAILED at 1928.
> 1 out of 1 hunk FAILED -- saving rejects to file tests/test_ssl.py.rej
> source is at 'pyOpenSSL-22.0.0'
> applying '/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch'...
> Backtrace:
> 
> [...]
> 
> ERROR:
>   1. &invoke-error:
>       program: "/gnu/store/z39hnrwds1dgcbpfgj8dnv2cngjb2xbl-patch-2.7.6/bin/patch"
>       arguments: ("--force" "--no-backup-if-mismatch" "-p1" "--input" "/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch")
> 
> [...]
> 
> builder for `/gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv' failed with exit code 1
> build of /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv failed
> View build log at '/var/log/guix/drvs/rf/awn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv.gz'.
> cannot build derivation `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv': 1 dependencies couldn't be built
> guix build: error: build of `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv' failed
> --8<---------------cut here---------------end--------------->8---
> 
> That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
> ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
> is that the ’origin’ also inherits
> 
>        (origin
>          (inherit (package-source base))
>          
> and thus
> 
>        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
> 
> does not make sense anymore because this patch is against 21.0.0 and
> your variant is about 22.0.0.
> 
> 
> Well, further investigations about what could be wrong for “guix pack”
> needs a way to reproduce. :-)  Could you fix or point a working example
> in your Git repository?
> 
> 
> Thanks,
> simon
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 9404 bytes --]

My problem has been solved. It turned out the Python interpreter
contained within the pack was finding an older version of `hydrilla`
Python package installed in `~/.local/lib/python3.9/site-packages` and
that older version was missing the `console_scripts` entry point that
was being loaded. It's worth mentioning that Python interpreter gives
`~/.local/lib/python3.9/site-packages` priority over the paths that
Guix adds to GUIX_PYTHONPATH.

The solution was to patch the wrapper script for each of the commands
my package provides. Definition of PYTHONNOUSERSITE enviroment variable
stops Python from looking at local site packages.

You can see my fix here[1]. I most importantly needed to add the
following to #:phases of my package definition

> (add-after 'wrap 'prevent-local-package-interference
>   (lambda* (#:key outputs #:allow-other-keys)
>     (match-let ((((_ . dir)) outputs))
>       (for-each (lambda (prog-name)
>                   (substitute* (string-append dir "/bin/" prog-name)
>                     (("^#!/.*$" shabang)
>                      (string-append shabang
>                                     "export PYTHONNOUSERSITE=1\n"))))
>                 '("hydrilla"
>                   "hydrilla-server"
>                   "hydrilla-builder"
>                   "haketilo")))))

It's worth noting that this problem is not exclusive to `guix pack` or
to my particular package. Users of other Python programs could in some
circumstances experience similar issues. Which makes me think -
shouldn't the default behavior be changed? Perhaps by making Python
give paths from `GUIX_PYTHONPATH` priority over those in user site
packages directory? Should I report this as a bug to bug-guix@gnu.org?

Best,
Wojtek

[1] https://git.koszko.org/pydrilla/commit/?h=koszko&id=f7c4dff95e88d58d61c51b2a6b94c90e2e4e0c4b


-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #17: blessed Jolenta Helena
Poznaj świętych krakowskich!  #17: błogosławiona Jolenta Helena
https://pl.wikipedia.org/wiki/Jolenta_Helena
-- (sig_end)


On Mon, 17 Oct 2022 15:36:48 +0200
Wojtek Kosior via <help-guix@gnu.org> wrote:

> Hello again after a break,
> 
> > Well, then I tried to reproduce your failure in order to understand what
> > is going wrong but I hit this:
> >
> > [...]
> > 
> > That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
> > ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
> > is that the ’origin’ also inherits
> > 
> >        (origin
> >          (inherit (package-source base))
> >          
> > and thus
> > 
> >        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
> > 
> > does not make sense anymore because this patch is against 21.0.0 and
> > your variant is about 22.0.0.  
> 
> Thank you for your time spent on trying this out.
> 
> The error is surely a result of us using different Guix versions. In the
> version I had the pyOpenSSL version was 20.0.1 and no patch was being
> used yet[1].
> 
> Another user also had problems building from source under a different
> Guix version. I knew this would happen but I didn't have a ready
> solution before. I now came up with a Makefile that organizes most
> common development tasks in my project into rules that call `guix
> time-machine` under the hood[2]. This should be a general solution for
> this kind of problems.
> 
> > Well, further investigations about what could be wrong for “guix pack”
> > needs a way to reproduce. :-)  Could you fix or point a working
> > example in your Git repository?  
> 
> As for troubleshooting the creation of Hydrilla 3.0-beta1 pack, one can
> get the same enviroment I was using by pulling Guix from commit
> fd00ac7e27c8a6d4b4d4719b4f8797bdce204623. Although most list users here
> probably know how to achieve this, I'm putting this here for convenience
> 
> > guix time-machine --commit=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623 -- <REST-OF-THE-USUAL-GUIX-COMMAND>  
> 
> Anyway, the main problem could probably be also troubleshooted by
> experimenting with the pack I made available[3] as it is also the one
> that is known to have caused problems.
> 
> Thank you again,
> Wojtek
> 
> [1] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-crypto.scm?id=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623#n650
> [2] https://git.koszko.org/pydrilla/commit/?h=koszko&id=ed2e14dd6ced4f64b6ac707c5998b38a78e6d7ad
> [3] https://hydrillabugs.koszko.org/projects/haketilo/wiki/Releases#Files
> 
> -- (sig_start)
> website: https://koszko.org/koszko.html
> PGP: https://koszko.org/key.gpg
> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
> 
> Meet Kraków saints!           #13: blessed Jan Beyzym
> Poznaj świętych krakowskich!  #13: błogosławiony Jan Beyzym
> https://pl.wikipedia.org/wiki/Jan_Beyzym
> -- (sig_end)
> 
> 
> On Fri, 14 Oct 2022 13:00:18 +0200
> zimoun <zimon.toutoune@gmail.com> wrote:
> 
> > Hi,
> > 
> > On ven., 14 oct. 2022 at 11:09, Wojtek Kosior via <help-guix@gnu.org> wrote:
> >   
> > > I accidently just replied to you, Simon, instead of making a "reply
> > > all". I'm reposting the same now, sorry for the nuisance...    
> > 
> > Do not worry. :-)
> >   
> > > So, I did run `guix shell -L. hydrilla`. First, I got a warning about
> > >    
> > >> ambiguous package specification `hydrilla'    
> > 
> > That’s expected because:
> > 
> > --8<---------------cut here---------------start------------->8---
> > (define-public hydrilla
> >   (package
> >     (name "hydrilla")
> > 
> > [...]
> > 
> > (define-public hydrilla-dist-tarball
> >   (let ((base hydrilla)
> >         (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
> >     (package
> >       (inherit base)
> >       (source (local-file
> > --8<---------------cut here---------------end--------------->8---
> > 
> > It means that the package ’hydrilla’ has the name “hydrilla” and because
> > the package ’hydrilla-dist-tarball’ inherits then it also has the name
> > “hydrilla”.  Idem for the version field.
> > 
> > Therefore,
> > 
> >     guix shell -L . hydrilla
> > 
> > refers to 2 possible packages.  All Guix CLI commands use “name“ and
> > “version” for looking up to packages and not the symbols.
> > 
> > 
> > Well, then I tried to reproduce your failure in order to understand what
> > is going wrong but I hit this:
> > 
> > --8<---------------cut here---------------start------------->8---
> > $ guix build -L . python-pyopenssl@22.0.0
> > The following derivations will be built:
> >   /gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv
> >   /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv
> > building /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv...
> > pyOpenSSL-22.0.0/
> > 
> > [...]
> > 
> > pyOpenSSL-22.0.0/tox.ini
> > patching file src/OpenSSL/SSL.py
> > Hunk #1 FAILED at 1421.
> > Hunk #2 FAILED at 2449.
> > 2 out of 2 hunks FAILED -- saving rejects to file src/OpenSSL/SSL.py.rej
> > patching file tests/test_ssl.py
> > Hunk #1 FAILED at 1928.
> > 1 out of 1 hunk FAILED -- saving rejects to file tests/test_ssl.py.rej
> > source is at 'pyOpenSSL-22.0.0'
> > applying '/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch'...
> > Backtrace:
> > 
> > [...]
> > 
> > ERROR:
> >   1. &invoke-error:
> >       program: "/gnu/store/z39hnrwds1dgcbpfgj8dnv2cngjb2xbl-patch-2.7.6/bin/patch"
> >       arguments: ("--force" "--no-backup-if-mismatch" "-p1" "--input" "/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch")
> > 
> > [...]
> > 
> > builder for `/gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv' failed with exit code 1
> > build of /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv failed
> > View build log at '/var/log/guix/drvs/rf/awn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv.gz'.
> > cannot build derivation `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv': 1 dependencies couldn't be built
> > guix build: error: build of `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv' failed
> > --8<---------------cut here---------------end--------------->8---
> > 
> > That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
> > ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
> > is that the ’origin’ also inherits
> > 
> >        (origin
> >          (inherit (package-source base))
> >          
> > and thus
> > 
> >        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
> > 
> > does not make sense anymore because this patch is against 21.0.0 and
> > your variant is about 22.0.0.
> > 
> > 
> > Well, further investigations about what could be wrong for “guix pack”
> > needs a way to reproduce. :-)  Could you fix or point a working example
> > in your Git repository?
> > 
> > 
> > Thanks,
> > simon
> >   
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[Csepp]

Wojtek Kosior via <help-guix@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> My problem has been solved. It turned out the Python interpreter
> contained within the pack was finding an older version of `hydrilla`
> Python package installed in `~/.local/lib/python3.9/site-packages` and
> that older version was missing the `console_scripts` entry point that
> was being loaded. It's worth mentioning that Python interpreter gives
> `~/.local/lib/python3.9/site-packages` priority over the paths that
> Guix adds to GUIX_PYTHONPATH.
>
> The solution was to patch the wrapper script for each of the commands
> my package provides. Definition of PYTHONNOUSERSITE enviroment variable
> stops Python from looking at local site packages.
>
> You can see my fix here[1]. I most importantly needed to add the
> following to #:phases of my package definition
>
>> (add-after 'wrap 'prevent-local-package-interference
>>   (lambda* (#:key outputs #:allow-other-keys)
>>     (match-let ((((_ . dir)) outputs))
>>       (for-each (lambda (prog-name)
>>                   (substitute* (string-append dir "/bin/" prog-name)
>>                     (("^#!/.*$" shabang)
>>                      (string-append shabang
>>                                     "export PYTHONNOUSERSITE=1\n"))))
>>                 '("hydrilla"
>>                   "hydrilla-server"
>>                   "hydrilla-builder"
>>                   "haketilo")))))
>
> It's worth noting that this problem is not exclusive to `guix pack` or
> to my particular package. Users of other Python programs could in some
> circumstances experience similar issues. Which makes me think -
> shouldn't the default behavior be changed? Perhaps by making Python
> give paths from `GUIX_PYTHONPATH` priority over those in user site
> packages directory? Should I report this as a bug to bug-guix@gnu.org?
>
> Best,
> Wojtek
>
> [1]
> https://git.koszko.org/pydrilla/commit/?h=koszko&id=f7c4dff95e88d58d61c51b2a6b94c90e2e4e0c4b
>
>
> -- (sig_start)
> website: https://koszko.org/koszko.html
> PGP: https://koszko.org/key.gpg
> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
>
> Meet Kraków saints!           #17: blessed Jolenta Helena
> Poznaj świętych krakowskich!  #17: błogosławiona Jolenta Helena
> https://pl.wikipedia.org/wiki/Jolenta_Helena
> -- (sig_end)
>
>
> On Mon, 17 Oct 2022 15:36:48 +0200
> Wojtek Kosior via <help-guix@gnu.org> wrote:
>
>> Hello again after a break,
>> 
>> > Well, then I tried to reproduce your failure in order to understand what
>> > is going wrong but I hit this:
>> >
>> > [...]
>> > 
>> > That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
>> > ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
>> > is that the ’origin’ also inherits
>> > 
>> >        (origin
>> >          (inherit (package-source base))
>> >          
>> > and thus
>> > 
>> >        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
>> > 
>> > does not make sense anymore because this patch is against 21.0.0 and
>> > your variant is about 22.0.0.  
>> 
>> Thank you for your time spent on trying this out.
>> 
>> The error is surely a result of us using different Guix versions. In the
>> version I had the pyOpenSSL version was 20.0.1 and no patch was being
>> used yet[1].
>> 
>> Another user also had problems building from source under a different
>> Guix version. I knew this would happen but I didn't have a ready
>> solution before. I now came up with a Makefile that organizes most
>> common development tasks in my project into rules that call `guix
>> time-machine` under the hood[2]. This should be a general solution for
>> this kind of problems.
>> 
>> > Well, further investigations about what could be wrong for “guix pack”
>> > needs a way to reproduce. :-)  Could you fix or point a working
>> > example in your Git repository?  
>> 
>> As for troubleshooting the creation of Hydrilla 3.0-beta1 pack, one can
>> get the same enviroment I was using by pulling Guix from commit
>> fd00ac7e27c8a6d4b4d4719b4f8797bdce204623. Although most list users here
>> probably know how to achieve this, I'm putting this here for convenience
>> 
>> > guix time-machine
>> > --commit=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623 --
>> > <REST-OF-THE-USUAL-GUIX-COMMAND>
>> 
>> Anyway, the main problem could probably be also troubleshooted by
>> experimenting with the pack I made available[3] as it is also the one
>> that is known to have caused problems.
>> 
>> Thank you again,
>> Wojtek
>> 
>> [1]
>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-crypto.scm?id=fd00ac7e27c8a6d4b4d4719b4f8797bdce204623#n650
>> [2]
>> https://git.koszko.org/pydrilla/commit/?h=koszko&id=ed2e14dd6ced4f64b6ac707c5998b38a78e6d7ad
>> [3] https://hydrillabugs.koszko.org/projects/haketilo/wiki/Releases#Files
>> 
>> -- (sig_start)
>> website: https://koszko.org/koszko.html
>> PGP: https://koszko.org/key.gpg
>> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
>> 
>> Meet Kraków saints!           #13: blessed Jan Beyzym
>> Poznaj świętych krakowskich!  #13: błogosławiony Jan Beyzym
>> https://pl.wikipedia.org/wiki/Jan_Beyzym
>> -- (sig_end)
>> 
>> 
>> On Fri, 14 Oct 2022 13:00:18 +0200
>> zimoun <zimon.toutoune@gmail.com> wrote:
>> 
>> > Hi,
>> > 
>> > On ven., 14 oct. 2022 at 11:09, Wojtek Kosior via <help-guix@gnu.org> wrote:
>> >   
>> > > I accidently just replied to you, Simon, instead of making a "reply
>> > > all". I'm reposting the same now, sorry for the nuisance...    
>> > 
>> > Do not worry. :-)
>> >   
>> > > So, I did run `guix shell -L. hydrilla`. First, I got a warning about
>> > >    
>> > >> ambiguous package specification `hydrilla'    
>> > 
>> > That’s expected because:
>> > 
>> > --8<---------------cut here---------------start------------->8---
>> > (define-public hydrilla
>> >   (package
>> >     (name "hydrilla")
>> > 
>> > [...]
>> > 
>> > (define-public hydrilla-dist-tarball
>> >   (let ((base hydrilla)
>> >         (filename (string-append "hydrilla-" %hydrilla-version ".tar.gz")))
>> >     (package
>> >       (inherit base)
>> >       (source (local-file
>> > --8<---------------cut here---------------end--------------->8---
>> > 
>> > It means that the package ’hydrilla’ has the name “hydrilla” and because
>> > the package ’hydrilla-dist-tarball’ inherits then it also has the name
>> > “hydrilla”.  Idem for the version field.
>> > 
>> > Therefore,
>> > 
>> >     guix shell -L . hydrilla
>> > 
>> > refers to 2 possible packages.  All Guix CLI commands use “name“ and
>> > “version” for looking up to packages and not the symbols.
>> > 
>> > 
>> > Well, then I tried to reproduce your failure in order to understand what
>> > is going wrong but I hit this:
>> > 
>> > --8<---------------cut here---------------start------------->8---
>> > $ guix build -L . python-pyopenssl@22.0.0
>> > The following derivations will be built:
>> >   /gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv
>> >   /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv
>> > building /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv...
>> > pyOpenSSL-22.0.0/
>> > 
>> > [...]
>> > 
>> > pyOpenSSL-22.0.0/tox.ini
>> > patching file src/OpenSSL/SSL.py
>> > Hunk #1 FAILED at 1421.
>> > Hunk #2 FAILED at 2449.
>> > 2 out of 2 hunks FAILED -- saving rejects to file src/OpenSSL/SSL.py.rej
>> > patching file tests/test_ssl.py
>> > Hunk #1 FAILED at 1928.
>> > 1 out of 1 hunk FAILED -- saving rejects to file tests/test_ssl.py.rej
>> > source is at 'pyOpenSSL-22.0.0'
>> > applying '/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch'...
>> > Backtrace:
>> > 
>> > [...]
>> > 
>> > ERROR:
>> >   1. &invoke-error:
>> >       program: "/gnu/store/z39hnrwds1dgcbpfgj8dnv2cngjb2xbl-patch-2.7.6/bin/patch"
>> >       arguments: ("--force" "--no-backup-if-mismatch" "-p1" "--input" "/gnu/store/4cd1x970w3jg3rwaiqg174x56mvkihij-python2-pyopenssl-openssl-compat.patch")
>> > 
>> > [...]
>> > 
>> > builder for `/gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv' failed with exit code 1
>> > build of /gnu/store/rfawn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv failed
>> > View build log at '/var/log/guix/drvs/rf/awn8bnr1vz71gcm6dz171gfpk0fwai-pyOpenSSL-22.0.0.tar.xz.drv.gz'.
>> > cannot build derivation `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv': 1 dependencies couldn't be built
>> > guix build: error: build of `/gnu/store/gyc6sw8a9vq09z6p7plv3k5sjjvrcz9i-python-pyopenssl-22.0.0.drv' failed
>> > --8<---------------cut here---------------end--------------->8---
>> > 
>> > That’s because your package ’python-pyopenssl-for-haketilo’ inherit from
>> > ’python-pyopenssl’ defined in (gnu packages python-crypto).  The issue
>> > is that the ’origin’ also inherits
>> > 
>> >        (origin
>> >          (inherit (package-source base))
>> >          
>> > and thus
>> > 
>> >        (patches (search-patches "python2-pyopenssl-openssl-compat.patch"))))
>> > 
>> > does not make sense anymore because this patch is against 21.0.0 and
>> > your variant is about 22.0.0.
>> > 
>> > 
>> > Well, further investigations about what could be wrong for “guix pack”
>> > needs a way to reproduce. :-)  Could you fix or point a working example
>> > in your Git repository?
>> > 
>> > 
>> > Thanks,
>> > simon
>> >   
>> 
>> 
>
>
> [[End of PGP Signed Part]]

IMHO yes, the pack output does not work as expected.  That's the
definition of a bug.

Re: program prepared with `guix pack` unusable by end users

[Maxim Cournoyer]

Hi,

Csepp <raingloom@riseup.net> writes:

> Wojtek Kosior via <help-guix@gnu.org> writes:
>
>> [[PGP Signed Part:Undecided]]
>> My problem has been solved. It turned out the Python interpreter
>> contained within the pack was finding an older version of `hydrilla`
>> Python package installed in `~/.local/lib/python3.9/site-packages` and
>> that older version was missing the `console_scripts` entry point that
>> was being loaded. It's worth mentioning that Python interpreter gives
>> `~/.local/lib/python3.9/site-packages` priority over the paths that
>> Guix adds to GUIX_PYTHONPATH.
>>
>> The solution was to patch the wrapper script for each of the commands
>> my package provides. Definition of PYTHONNOUSERSITE enviroment variable
>> stops Python from looking at local site packages.

[...]

>> It's worth noting that this problem is not exclusive to `guix pack` or
>> to my particular package. Users of other Python programs could in some
>> circumstances experience similar issues. Which makes me think -
>> shouldn't the default behavior be changed? Perhaps by making Python
>> give paths from `GUIX_PYTHONPATH` priority over those in user site
>> packages directory? Should I report this as a bug to bug-guix@gnu.org?
>>
>> Best,
>> Wojtek

[...]

> IMHO yes, the pack output does not work as expected.  That's the
> definition of a bug.

I disagree.  That Python gives precedence to USERSITE compared to
site-packages and GUIX_PYTHONPATH is by design, so that users can
override system provided libraries such as those by Guix.  It used to be
the other way around, and it caused all sort of problems such as
virtualenv not working as expected on Guix.

-- 
Thanks,
Maxim

Re: program prepared with `guix pack` unusable by end users

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 3433 bytes --]

> > IMHO yes, the pack output does not work as expected.  That's the
> > definition of a bug.  
> 
> I disagree.  That Python gives precedence to USERSITE compared to
> site-packages and GUIX_PYTHONPATH is by design, so that users can
> override system provided libraries such as those by Guix.  It used to be
> the other way around, and it caused all sort of problems such as
> virtualenv not working as expected on Guix.

Perhaps the best solution would be to
* have Python interpreter itself give precedence to user site packages but
* have user site disabled (or enabled with lower precedence) by default
  for Python applications.

Consider the creation of wrapper script for python programs as it is
done now[1]. Is there currently any application that would behave
incorrectly with PYTHONNOUSERSITE exported as 1 and
`~/.local/lib/python<PYTHON-VERSION>/site-packages/` included in
GUIX-PYTHONPATH after the other paths? If there is, perhaps it would be
at least easier to make a workaround for this single application?

[1] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/python-build-system.scm?id=176a501360699581b49f19ffde1ea3bb6285b8be#n225

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

Meet Kraków saints!           #0: saint Albert Chmielowski
Poznaj świętych krakowskich!  #0: święty Albert Chmielowski
https://pl.wikipedia.org/wiki/Adam_Chmielowski
-- (sig_end)


On Thu, 27 Oct 2022 12:59:23 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> Hi,
> 
> Csepp <raingloom@riseup.net> writes:
> 
> > Wojtek Kosior via <help-guix@gnu.org> writes:
> >  
> >> [[PGP Signed Part:Undecided]]
> >> My problem has been solved. It turned out the Python interpreter
> >> contained within the pack was finding an older version of `hydrilla`
> >> Python package installed in `~/.local/lib/python3.9/site-packages` and
> >> that older version was missing the `console_scripts` entry point that
> >> was being loaded. It's worth mentioning that Python interpreter gives
> >> `~/.local/lib/python3.9/site-packages` priority over the paths that
> >> Guix adds to GUIX_PYTHONPATH.
> >>
> >> The solution was to patch the wrapper script for each of the commands
> >> my package provides. Definition of PYTHONNOUSERSITE enviroment variable
> >> stops Python from looking at local site packages.  
> 
> [...]
> 
> >> It's worth noting that this problem is not exclusive to `guix pack` or
> >> to my particular package. Users of other Python programs could in some
> >> circumstances experience similar issues. Which makes me think -
> >> shouldn't the default behavior be changed? Perhaps by making Python
> >> give paths from `GUIX_PYTHONPATH` priority over those in user site
> >> packages directory? Should I report this as a bug to bug-guix@gnu.org?
> >>
> >> Best,
> >> Wojtek  
> 
> [...]
> 
> > IMHO yes, the pack output does not work as expected.  That's the
> > definition of a bug.  
> 
> I disagree.  That Python gives precedence to USERSITE compared to
> site-packages and GUIX_PYTHONPATH is by design, so that users can
> override system provided libraries such as those by Guix.  It used to be
> the other way around, and it caused all sort of problems such as
> virtualenv not working as expected on Guix.
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: program prepared with `guix pack` unusable by end users

[Maxim Cournoyer]

Hello,

Wojtek Kosior <koszko@koszko.org> writes:

>> > IMHO yes, the pack output does not work as expected.  That's the
>> > definition of a bug.  
>> 
>> I disagree.  That Python gives precedence to USERSITE compared to
>> site-packages and GUIX_PYTHONPATH is by design, so that users can
>> override system provided libraries such as those by Guix.  It used to be
>> the other way around, and it caused all sort of problems such as
>> virtualenv not working as expected on Guix.

My memory failed me here; the past problems with Guix and virtualenv had
to do with us (ab)using PYTHONPATH, which would take precedence above
everything and not get reset by virtualenv, if I remember.

--8<---------------cut here---------------start------------->8---
$ . bin/activate
(venv) maxim@hurd /tmp/venv [env]$ python -m site
sys.path = [
    '/tmp/venv',
    '/gnu/store/bbakd2h7s4g62fymjmqpxgvlanp1ydny-python-3.9.9/lib/python39.zip',
    '/gnu/store/bbakd2h7s4g62fymjmqpxgvlanp1ydny-python-3.9.9/lib/python3.9',
    '/gnu/store/bbakd2h7s4g62fymjmqpxgvlanp1ydny-python-3.9.9/lib/python3.9/lib-dynload',
    '/tmp/venv/lib/python3.9/site-packages',
]
USER_BASE: '/home/maxim/.local' (exists)
USER_SITE: '/home/maxim/.local/lib/python3.9/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
--8<---------------cut here---------------end--------------->8---

We can see that virtualenv disables loading libraries from USER_SITE
(ENABLE_USER_SITE: False).  We can also see that it puts its
site-packages directly on sys.path.

> Perhaps the best solution would be to
> * have Python interpreter itself give precedence to user site packages but
> * have user site disabled (or enabled with lower precedence) by default
>   for Python applications.
>
> Consider the creation of wrapper script for python programs as it is
> done now[1]. Is there currently any application that would behave
> incorrectly with PYTHONNOUSERSITE exported as 1 and
> `~/.local/lib/python<PYTHON-VERSION>/site-packages/` included in
> GUIX-PYTHONPATH after the other paths? If there is, perhaps it would be
> at least easier to make a workaround for this single application?

I agree that setting PYTHONNOUSERSITE to 1 in the wrapper of Python
commands to ensure no stray user-installed librairies from under
"~/.local/lib/python<PYTHON-VERSION>/site-packages/" clash with their
operation would make sense.

Note that this is an interoperability problem between 'guix' and 'pip',
for 'pip install x' users that end up filling their
"~/.local/lib/python<PYTHON-VERSION>/site-packages/" directory.  If you
'rm -rf ~/.local/lib/python*' and stick to use only Guix-provided
packages, they won't experience any issue.

-- 
Thanks,
Maxim