bug#56373: Updating synapse (Matrix Homeserver) Because it is Broken

bug#56373: Updating synapse (Matrix Homeserver) Because it is Broken

[Zacchaeus Scheffer]

[-- Attachment #1: Type: text/plain, Size: 8050 bytes --]

Hi Guix!


I'm trying to update synapse because it seems an update somewhere has
broken synapse (I'm thinking python -> 3.9.*?).  Specifically, I get the
following traceback:


$ synctl start .config/synapse/homeserver.yaml --no-daemonize
Starting ...
Traceback (most recent call last):
  File
"/gnu/store/z561ps804hs0shwicdw076wwg4mim8ml-python-3.9.9/lib/python3.9/runpy.py",
line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File
"/gnu/store/z561ps804hs0shwicdw076wwg4mim8ml-python-3.9.9/lib/python3.9/runpy.py",
line 87, in _run_code
    exec(code, run_globals)
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/app/homeserver.py",
line 45, in <module>
    from synapse.federation.transport.server import TransportLayerServer
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/federation/transport/server.py",
line 46, in <module>
    from synapse.server import HomeServer
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/server.py",
line 55, in <module>
    from synapse.events.spamcheck import SpamChecker
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/events/spamcheck.py",
line 20, in <module>
    from synapse.rest.media.v1._base import FileInfo
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/rest/__init__.py",
line 32, in <module>
    from synapse.rest.client.v2_alpha import (
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/rest/client/v2_alpha/account.py",
line 40, in <module>
    from synapse.push.mailer import Mailer
  File
"/gnu/store/sgxq33kpvbc56j1ag0rwhj3f9w16wfz0-synapse-1.29.0/lib/python3.9/site-packages/synapse/push/mailer.py",
line 860, in <module>
    def safe_markup(raw_html: str) -> jinja2.Markup:
AttributeError: module 'jinja2' has no attribute 'Markup'
error starting (exit code: 1); see above for logs


No logs are saved because the code crashes before it can get to that.  A
quick search brought me to
https://github.com/YunoHost-Apps/synapse_ynh/issues/304 which indicates
that the problem is expected and fixed in later versions.  It seems matrix
abstracted some code to a new repository, as just increasing the synapse
version number causes it to fail as it looks for a library:
"matrix-common".  My current working definition for matrix-common is:


(define-public python-matrix-common
  (package
    (name "python-matrix-common")
    (version "1.2.0") ; tried 1.2.1 and 1.2.0
    (source (origin
              (method url-fetch)
              (uri (pypi-uri "matrix_common" version))
              (sha256
               (base32
                "0lrqzb6s57fxp0kwffdqnkr2pj9aia459cv1b95b55dxlq1cz7d9"))))
                ;"1bgdhzvqs51z079zjszhd5xqb100mbr5w8gpxs9z31r5xmi5nw7a"))))
    (build-system python-build-system)
    (arguments
     `(#:use-setuptools? #f ; tried with and without this
       #:phases
       (modify-phases %standard-phases
         (replace 'build
           (lambda _
             (setenv "SOURCE_DATE_EPOCH" "315532800")
             (invoke "python" "-m" "build" "--wheel" "--no-isolation" ".")))
         (delete 'check)
         (replace 'install
           (lambda* (#:key outputs #:allow-other-keys)
             (let ((out (assoc-ref outputs "out"))
                   (whl (car (find-files "dist" "\\.whl$"))))
               (invoke "pip" "--no-cache-dir" "--no-input"
                       "install" "--no-deps" "--prefix" out whl))))
         (delete 'sanity-check))))
    (propagated-inputs
     (list))
    (native-inputs
     (list python-pypa-build
           python-attrs))
    (home-page "https://github.com/matrix-org/matrix-python-common")
    (synopsis "Common utilities for Synapse, Sydent and Sygnal")
    (description "")
    (license license:asl2.0)))


The matrix-common library has no setup.py, hence I tried replacing the
build and install phase with something similar to what was done for
python-isort as suggested in the irc.  I deleted check for now because it
was looking for setup.py.  This definition builds without printing errors.
However, when I try to build synapse in v1.61.1 (adding
python-matrix-common to the native-inputs), I get the following output:


starting phase `check'
running "python setup.py" with command "test" and parameters ()
running test
WARNING: Testing via this command is deprecated and will be removed in a
future version. Users looking for a generic test entry point independent of
test runner are encouraged to use tox.
WARNING: The wheel package is not available.
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent
directory is not owned or is not writable by the current user. The cache
has been disabled. Check the permissions and owner of that directory. If
executing pip with sudo, you should use sudo's -H flag.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff59da6d0>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/matrix-common/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff59da610>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/matrix-common/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff59da040>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/matrix-common/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff59fc070>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/matrix-common/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None,
status=None)) after connection broken by
'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object
at 0x7ffff59fc220>: Failed to establish a new connection: [Errno -2] Name
or service not known')': /simple/matrix-common/
ERROR: Could not find a version that satisfies the requirement
matrix-common<1.2.0,>=1.1.0 (from versions: none)
ERROR: No matching distribution found for matrix-common<1.2.0,>=1.1.0
error: Command
'['/gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python',
'-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w',
'/tmp/guix-build-synapse-1.61.1.drv-0/tmphzjgsvuo', '--quiet',
'matrix-common<1.2.0,>=1.1.0']' returned non-zero exit status 1.
error: in phase 'check': uncaught exception:
%exception #<&invoke-error program: "python" arguments: ("-c" "import
setuptools, tokenize;__file__='setup.py';f=getattr(tokenize, 'open',
open)(__file__);code=f.read().replace('\\r\\n',
'\\n');f.close();exec(compile(code, __file__, 'exec'))" "test")
exit-status: 1 term-signal: #f stop-signal: #f>
phase `check' failed after 8.3 seconds
command "python" "-c" "import setuptools,
tokenize;__file__='setup.py';f=getattr(tokenize, 'open',
open)(__file__);code=f.read().replace('\\r\\n',
'\\n');f.close();exec(compile(code, __file__, 'exec'))" "test" failed with
status 1
builder for
`/gnu/store/n0s40wdrkc8sgdlkdv67d4y23yl7acir-synapse-1.61.1.drv' failed
with exit code 1


So it looks like the library I packaged (python-matrix-common) is not being
found.  Anyone more experienced with packaging python have any ideas?  The
matrix-common repository can be found at
https://github.com/matrix-org/matrix-python-common.


Thanks,
Zacchaeus

[-- Attachment #2: Type: text/html, Size: 9683 bytes --]

bug#56373: Updating synapse (Matrix Homeserver) Because it is Broken

[Michael Rohleder]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

Hi Zacchaeus!

Thanks for the report!

I think, synapse 1.30.1 changed the dependency of python-cryptography to
a newer version (which we currently have as python-cryptography-next).
But also python-pyopenssl and python-pysaml2 needs to be build with
cryptography-next [1] (or they fail in a byte2int() or similar at runtime
somewhere).

So we also need some more -next python packages to upgrade synapse from
1.29.

matrix-common is needed from version >=1.50.


Footnotes:
[1]  http://www.rohleder.de/gitweb/?p=guix.git;a=blob;f=mroh/guix/packages/python.scm;h=13ce4e8dd9822f443633385af6203fef78e198aa;hb=HEAD#l79
-- 
There are two ways of constructing a software design.
One is to make it so simple that there are obviously no deficiencies;
the other is to make it so complicated that there are no obvious deficiencies.
The first method is far more difficult.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]