From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:42278) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j6ApO-0005aY-F8 for guix-patches@gnu.org; Mon, 24 Feb 2020 05:19:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j6ApK-0001Kc-J7 for guix-patches@gnu.org; Mon, 24 Feb 2020 05:19:06 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:46945) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j6ApK-0001KU-CZ for guix-patches@gnu.org; Mon, 24 Feb 2020 05:19:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1j6ApK-0005J9-6c for guix-patches@gnu.org; Mon, 24 Feb 2020 05:19:02 -0500 Subject: [bug#39765] Add package JupyterLab Resent-Message-ID: Received: from eggs.gnu.org ([2001:470:142:3::10]:42146) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j6AoZ-0005GB-UA for guix-patches@gnu.org; Mon, 24 Feb 2020 05:18:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j6AoV-00019L-Og for guix-patches@gnu.org; Mon, 24 Feb 2020 05:18:15 -0500 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:37668) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j6AoU-00018y-Vb for guix-patches@gnu.org; Mon, 24 Feb 2020 05:18:11 -0500 Received: by mail-wr1-x433.google.com with SMTP id l5so5423743wrx.4 for ; Mon, 24 Feb 2020 02:18:10 -0800 (PST) Received: from localhost (zpidlx9.uni-trier.de. [136.199.85.49]) by smtp.gmail.com with ESMTPSA id z133sm18024672wmb.7.2020.02.24.02.18.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Feb 2020 02:18:08 -0800 (PST) Date: Mon, 24 Feb 2020 11:18:10 +0100 From: Lars-Dominik Braun Message-ID: <20200224101810.GA9010@zpidnp36> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="sdtB3X0nJg68CQEu" Content-Disposition: inline Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 39765@debbugs.gnu.org --sdtB3X0nJg68CQEu Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi, this patch series adds Jupyter’s JupyterLab, which is the new frontend for Jupyter Notebooks. The software works fine, but there are a few caveats 1) it comes with bundled pre-compiled JavaScript, which cannot be removed until we have proper support for importing from NPM 2) it contains an extension manager, that downloads arbitrary packages from NPM (`jupyter lab build`). This works, but is less than optimal imo. We should figure out how to package extensions in guix. 3) also it is required to install the package `jupyter`, otherwise installed kernels cannot be found and the `jupyter` command does not work. Cheers, Lars --sdtB3X0nJg68CQEu Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-gnu-Add-package-python-pytest-check-links.patch" >From 4a5862e2add1d537770a5ea466dbb8a4851afad9 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 7 Feb 2020 08:38:32 +0100 Subject: [PATCH 1/5] gnu: Add package python-pytest-check-links * gnu/packages/python-xyz.scm (python-pytest-check-links): New variable. --- gnu/packages/python-xyz.scm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 84b70954bd..902ca5030b 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -17570,3 +17570,31 @@ sequences.") (define-public python2-fuzzywuzzy (package-with-python2 python-fuzzywuzzy)) + +(define-public python-pytest-check-links + (package + (name "python-pytest-check-links") + (version "0.3.0") + (source + (origin + (method url-fetch) + ;; URI uses underscores + (uri (pypi-uri "pytest_check_links" version)) + (sha256 + (base32 + "12x3wmrdzm6wgk0vz02hb769h68nr49q47w5q1pj95pc89hsa34v")))) + (build-system python-build-system) + (propagated-inputs + `(("python-docutils" ,python-docutils) + ("python-html5lib" ,python-html5lib) + ("python-nbconvert" ,python-nbconvert) + ("python-nbformat" ,python-nbformat) + ("python-pytest" ,python-pytest) + ("python-six" ,python-six))) + (native-inputs + `(("python-pbr-minimal" ,python-pbr-minimal))) + (home-page + "https://github.com/minrk/pytest-check-links") + (synopsis "Check links in files") + (description "Plugin for pytest that checks URLs for HTML-containing files") + (license license:bsd-3))) -- 2.20.1 --sdtB3X0nJg68CQEu Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-gnu-Add-package-python-json5.patch" >From 690d45cba7d1a21fa6ae97fbe4fec6e1abae3635 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 7 Feb 2020 08:39:55 +0100 Subject: [PATCH 2/5] gnu: Add package python-json5 * gnu/packages/python-xyz.scm (python-json5): New variable. --- gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 902ca5030b..ad1cd5fbd4 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -17598,3 +17598,27 @@ sequences.") (synopsis "Check links in files") (description "Plugin for pytest that checks URLs for HTML-containing files") (license license:bsd-3))) + +(define-public python-json5 + (package + (name "python-json5") + (version "0.8.5") + (source + (origin + ;; sample.json5 is missing from PyPi source tarball + (method git-fetch) + (uri (git-reference + (url "https://github.com/dpranke/pyjson5.git") + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 "0nyngj18jlkgvm1177lc3cj47wm4yh3dqigygvcvw7xkyryafsqn")))) + (build-system python-build-system) + (home-page "https://github.com/dpranke/pyjson5") + (synopsis + "Python implementation of the JSON5 data format") + (description + "JSON5 extends the JSON data interchange format to make it slightly more +usable as a configuration language. This Python package implements parsing and +dumping of JSON5 data structures.") + (license license:asl2.0))) -- 2.20.1 --sdtB3X0nJg68CQEu Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-gnu-Add-package-python-jupyterlab-server.patch" >From a4535f6002a618444171d5a92146fb7a7e7e8243 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 7 Feb 2020 08:40:41 +0100 Subject: [PATCH 3/5] gnu: Add package python-jupyterlab-server * gnu/packages/python-xyz.scm (python-jupyterlab-server): New variable. --- gnu/packages/python-xyz.scm | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index ad1cd5fbd4..92ee53fe6f 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -17622,3 +17622,37 @@ sequences.") usable as a configuration language. This Python package implements parsing and dumping of JSON5 data structures.") (license license:asl2.0))) + +(define-public python-jupyterlab-server + (package + (name "python-jupyterlab-server") + (version "1.0.6") + (source + (origin + (method url-fetch) + (uri (pypi-uri "jupyterlab_server" version)) + (sha256 + (base32 + "1bax8iqwcc5p02h5ysdc48zvx7ll5jfzfsybhb3lfvyfpwkpb5yh")))) + (build-system python-build-system) + (propagated-inputs + `(("python-jinja2" ,python-jinja2) + ("python-json5" ,python-json5) + ("python-jsonschema" ,python-jsonschema) + ("python-notebook" ,python-notebook))) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-requests" ,python-requests) + ("python-ipykernel" ,python-ipykernel))) + (arguments + `(#:phases + (modify-phases %standard-phases + ;; python setup.py test does not invoke pytest? + (replace 'check + (lambda _ + (invoke "pytest" "-vv")))))) + (home-page "https://jupyter.org") + (synopsis "JupyterLab Server") + (description "A set of server components for JupyterLab and JupyterLab like +applications") + (license license:bsd-3))) -- 2.20.1 --sdtB3X0nJg68CQEu Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0004-gnu-Add-package-python-jupyterlab.patch" Content-Transfer-Encoding: 8bit >From c50fc3ec734cb94e78168a4a29c9aff070f4ae9f Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 7 Feb 2020 08:41:24 +0100 Subject: [PATCH 4/5] gnu: Add package python-jupyterlab * gnu/packages/python-xyz.scm (python-jupyterlab): New variable. --- .../python-jupyterlab-copy-nometa.patch | 33 +++++++++++ gnu/packages/python-xyz.scm | 55 +++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 gnu/packages/patches/python-jupyterlab-copy-nometa.patch diff --git a/gnu/packages/patches/python-jupyterlab-copy-nometa.patch b/gnu/packages/patches/python-jupyterlab-copy-nometa.patch new file mode 100644 index 0000000000..5e770c0f47 --- /dev/null +++ b/gnu/packages/patches/python-jupyterlab-copy-nometa.patch @@ -0,0 +1,33 @@ +diff '--exclude=*.swp' -Naur jupyterlab-1.2.6.orig/jupyterlab/commands.py jupyterlab-1.2.6/jupyterlab/commands.py +--- jupyterlab-1.2.6.orig/jupyterlab/commands.py 2020-01-24 17:25:16.238353500 +0100 ++++ jupyterlab-1.2.6/jupyterlab/commands.py 2020-01-31 12:36:10.497375982 +0100 +@@ -1100,7 +1100,7 @@ + 'webpack.prod.minimize.config.js', + '.yarnrc', 'yarn.js']: + target = pjoin(staging, fname) +- shutil.copy(pjoin(HERE, 'staging', fname), target) ++ shutil.copyfile(pjoin(HERE, 'staging', fname), target) + + # Ensure a clean templates directory + templates = pjoin(staging, 'templates') +@@ -1108,7 +1108,10 @@ + _rmtree(templates, self.logger) + + try: +- shutil.copytree(pjoin(HERE, 'staging', 'templates'), templates) ++ shutil.copytree(pjoin(HERE, 'staging', 'templates'), templates, ++ copy_function=shutil.copyfile) ++ # cannot replace or disable copytree’s call to .copystat ++ os.chmod (templates, 0o755) + except shutil.Error as error: + # `copytree` throws an error if copying to + from NFS even though + # the copy is successful (see https://bugs.python.org/issue24564 +@@ -1178,7 +1181,7 @@ + with open(lock_path, 'w', encoding='utf-8') as f: + f.write(template) + elif not osp.exists(lock_path): +- shutil.copy(lock_template, lock_path) ++ shutil.copyfile(lock_template, lock_path) + + def _get_package_template(self, silent=False): + """Get the template the for staging package.json file. diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 92ee53fe6f..232841ccb1 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -126,6 +126,7 @@ #:use-module (gnu packages multiprecision) #:use-module (gnu packages networking) #:use-module (gnu packages ncurses) + #:use-module (gnu packages node) #:use-module (gnu packages openstack) #:use-module (gnu packages pcre) #:use-module (gnu packages perl) @@ -17656,3 +17657,57 @@ dumping of JSON5 data structures.") (description "A set of server components for JupyterLab and JupyterLab like applications") (license license:bsd-3))) + +(define-public python-jupyterlab + (package + (name "python-jupyterlab") + (version "1.2.6") + (source + (origin + (method url-fetch) + (uri (pypi-uri "jupyterlab" version)) + (sha256 + (base32 + "0mc3nrj7fc5q2ajr09m261j386jsp8qjljg8anghlh8czc9ln4s2")) + (patches (search-patches "python-jupyterlab-copy-nometa.patch")))) + (build-system python-build-system) + (propagated-inputs + `(("python-jinja2" ,python-jinja2) + ("python-jupyterlab-server" + ,python-jupyterlab-server) + ("python-notebook" ,python-notebook) + ("python-tornado" ,python-tornado) + ("node" ,node))) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-pytest-check-links" + ,python-pytest-check-links) + ("python-requests" ,python-requests) + ("python-ipykernel" ,python-ipykernel))) + (arguments + ;; testing requires npm, so disabled for now + '(#:tests? #f + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-syspath + (lambda* (#:key outputs inputs configure-flags #:allow-other-keys) + (let* ((out (assoc-ref outputs "out"))) + (substitute* "jupyterlab/commands.py" + ;; sys.prefix defaults to Python’s prefix in the store, not + ;; jupyterlab’s. Fix that. + (("sys\\.prefix") + (string-append "'" out "'")))) + #t)) + ;; 'build does not respect configure-flags + (replace 'build + (lambda _ + (invoke "python" "setup.py" "build" "--skip-npm")))) + #:configure-flags (list "--skip-npm"))) + (home-page "https://jupyter.org") + (synopsis + "The JupyterLab notebook server extension") + (description + "An extensible environment for interactive and reproducible computing, +based on the Jupyter Notebook and Architecture.") + (license license:bsd-3))) + -- 2.20.1 --sdtB3X0nJg68CQEu Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-gnu-python-notebook-Support-UNIX-domain-sockets.patch" >From a47fd94aa6f3e62b77f3b7208c4e6757e3a9ee08 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Thu, 12 Dec 2019 08:53:39 +0100 Subject: [PATCH 5/5] gnu: python-notebook: Support UNIX domain sockets * gnu/packages/python-xyz.scm (python-notebook): Add patch from upstream https://github.com/jupyter/notebook/pull/4835 (python-requests-unixsocket) New variable --- ...pyter-unix-domain-sockets-4835-5.7.4.patch | 591 ++++++++++++++++++ gnu/packages/python-xyz.scm | 35 +- 2 files changed, 624 insertions(+), 2 deletions(-) create mode 100644 gnu/packages/patches/jupyter-unix-domain-sockets-4835-5.7.4.patch diff --git a/gnu/packages/patches/jupyter-unix-domain-sockets-4835-5.7.4.patch b/gnu/packages/patches/jupyter-unix-domain-sockets-4835-5.7.4.patch new file mode 100644 index 0000000000..134d3ad2b8 --- /dev/null +++ b/gnu/packages/patches/jupyter-unix-domain-sockets-4835-5.7.4.patch @@ -0,0 +1,591 @@ +diff -Naur notebook-5.7.4/notebook/base/handlers.py notebook-5.7.4.patched/notebook/base/handlers.py +--- notebook-5.7.4/notebook/base/handlers.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/base/handlers.py 2019-11-18 12:16:58.315065024 +0100 +@@ -40,7 +40,7 @@ + import notebook + from notebook._tz import utcnow + from notebook.i18n import combine_translations +-from notebook.utils import is_hidden, url_path_join, url_is_absolute, url_escape ++from notebook.utils import is_hidden, url_path_join, url_is_absolute, url_escape, urldecode_unix_socket_path + from notebook.services.security import csp_report_uri + + #----------------------------------------------------------------------------- +@@ -426,13 +426,18 @@ + # ip_address only accepts unicode on Python 2 + host = host.decode('utf8', 'replace') + +- try: +- addr = ipaddress.ip_address(host) +- except ValueError: +- # Not an IP address: check against hostnames +- allow = host in self.settings.get('local_hostnames', ['localhost']) ++ # UNIX socket handling ++ check_host = urldecode_unix_socket_path(host) ++ if check_host.startswith('/') and os.path.exists(check_host): ++ allow = True + else: +- allow = addr.is_loopback ++ try: ++ addr = ipaddress.ip_address(host) ++ except ValueError: ++ # Not an IP address: check against hostnames ++ allow = host in self.settings.get('local_hostnames', ['localhost']) ++ else: ++ allow = addr.is_loopback + + if not allow: + self.log.warning( +diff -Naur notebook-5.7.4/notebook/__init__.py notebook-5.7.4.patched/notebook/__init__.py +--- notebook-5.7.4/notebook/__init__.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/__init__.py 2019-11-18 12:16:58.315065024 +0100 +@@ -20,6 +20,8 @@ + os.path.join(os.path.dirname(__file__), "templates"), + ] + ++DEFAULT_NOTEBOOK_PORT = 8888 ++ + del os + + from .nbextensions import install_nbextension +diff -Naur notebook-5.7.4/notebook/notebookapp.py notebook-5.7.4.patched/notebook/notebookapp.py +--- notebook-5.7.4/notebook/notebookapp.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/notebookapp.py 2019-11-18 12:21:34.975072928 +0100 +@@ -63,8 +63,11 @@ + from tornado import web + from tornado.httputil import url_concat + from tornado.log import LogFormatter, app_log, access_log, gen_log ++if not sys.platform.startswith('win'): ++ from tornado.netutil import bind_unix_socket + + from notebook import ( ++ DEFAULT_NOTEBOOK_PORT, + DEFAULT_STATIC_FILES_PATH, + DEFAULT_TEMPLATE_PATH_LIST, + __version__, +@@ -108,7 +111,16 @@ + from notebook._sysinfo import get_sys_info + + from ._tz import utcnow, utcfromtimestamp +-from .utils import url_path_join, check_pid, url_escape, urljoin, pathname2url ++from .utils import ( ++ check_pid, ++ pathname2url, ++ url_escape, ++ url_path_join, ++ urldecode_unix_socket_path, ++ urlencode_unix_socket, ++ urlencode_unix_socket_path, ++ urljoin, ++) + + #----------------------------------------------------------------------------- + # Module globals +@@ -212,7 +224,7 @@ + warnings.warn(_("The `ignore_minified_js` flag is deprecated and will be removed in Notebook 6.0"), DeprecationWarning) + + now = utcnow() +- ++ + root_dir = contents_manager.root_dir + home = os.path.expanduser('~') + if root_dir.startswith(home + os.path.sep): +@@ -385,6 +397,7 @@ + set_password(config_file=self.config_file) + self.log.info("Wrote hashed password to %s" % self.config_file) + ++ + def shutdown_server(server_info, timeout=5, log=None): + """Shutdown a notebook server in a separate process. + +@@ -397,14 +410,39 @@ + Returns True if the server was stopped by any means, False if stopping it + failed (on Windows). + """ +- from tornado.httpclient import HTTPClient, HTTPRequest ++ from tornado import gen ++ from tornado.httpclient import AsyncHTTPClient, HTTPClient, HTTPRequest ++ from tornado.netutil import bind_unix_socket, Resolver + url = server_info['url'] + pid = server_info['pid'] ++ resolver = None ++ ++ # UNIX Socket handling. ++ if url.startswith('http+unix://'): ++ # This library doesn't understand our URI form, but it's just HTTP. ++ url = url.replace('http+unix://', 'http://') ++ ++ class UnixSocketResolver(Resolver): ++ def initialize(self, resolver): ++ self.resolver = resolver ++ ++ def close(self): ++ self.resolver.close() ++ ++ @gen.coroutine ++ def resolve(self, host, port, *args, **kwargs): ++ raise gen.Return([ ++ (socket.AF_UNIX, urldecode_unix_socket_path(host)) ++ ]) ++ ++ resolver = UnixSocketResolver(resolver=Resolver()) ++ + req = HTTPRequest(url + 'api/shutdown', method='POST', body=b'', headers={ + 'Authorization': 'token ' + server_info['token'] + }) + if log: log.debug("POST request to %sapi/shutdown", url) +- HTTPClient().fetch(req) ++ AsyncHTTPClient.configure(None, resolver=resolver) ++ HTTPClient(AsyncHTTPClient).fetch(req) + + # Poll to see if it shut down. + for _ in range(timeout*10): +@@ -435,13 +473,20 @@ + version = __version__ + description="Stop currently running notebook server for a given port" + +- port = Integer(8888, config=True, +- help="Port of the server to be killed. Default 8888") ++ port = Integer(DEFAULT_NOTEBOOK_PORT, config=True, ++ help="Port of the server to be killed. Default %s" % DEFAULT_NOTEBOOK_PORT) ++ ++ sock = Unicode(u'', config=True, ++ help="UNIX socket of the server to be killed.") + + def parse_command_line(self, argv=None): + super(NbserverStopApp, self).parse_command_line(argv) + if self.extra_args: +- self.port=int(self.extra_args[0]) ++ try: ++ self.port = int(self.extra_args[0]) ++ except ValueError: ++ # self.extra_args[0] was not an int, so it must be a string (unix socket). ++ self.sock = self.extra_args[0] + + def shutdown_server(self, server): + return shutdown_server(server, log=self.log) +@@ -451,16 +496,16 @@ + if not servers: + self.exit("There are no running servers") + for server in servers: +- if server['port'] == self.port: +- print("Shutting down server on port", self.port, "...") ++ if server.get('sock') == self.sock or server['port'] == self.port: ++ print("Shutting down server on %s..." % self.sock or self.port) + if not self.shutdown_server(server): + sys.exit("Could not stop server") + return + else: + print("There is currently no server running on port {}".format(self.port), file=sys.stderr) +- print("Ports currently in use:", file=sys.stderr) ++ print("Ports/sockets currently in use:", file=sys.stderr) + for server in servers: +- print(" - {}".format(server['port']), file=sys.stderr) ++ print(" - {}".format(server.get('sock', server['port'])), file=sys.stderr) + self.exit(1) + + +@@ -540,6 +585,8 @@ + 'ip': 'NotebookApp.ip', + 'port': 'NotebookApp.port', + 'port-retries': 'NotebookApp.port_retries', ++ 'sock': 'NotebookApp.sock', ++ 'sock-umask': 'NotebookApp.sock_umask', + 'transport': 'KernelManager.transport', + 'keyfile': 'NotebookApp.keyfile', + 'certfile': 'NotebookApp.certfile', +@@ -678,10 +725,18 @@ + or containerized setups for example).""") + ) + +- port = Integer(8888, config=True, ++ port = Integer(DEFAULT_NOTEBOOK_PORT, config=True, + help=_("The port the notebook server will listen on.") + ) + ++ sock = Unicode(u'', config=True, ++ help=_("The UNIX socket the notebook server will listen on.") ++ ) ++ ++ sock_umask = Unicode(u'0600', config=True, ++ help=_("The UNIX socket umask to set on creation (default: 0600).") ++ ) ++ + port_retries = Integer(50, config=True, + help=_("The number of additional ports to try if the specified port is not available.") + ) +@@ -1370,6 +1425,27 @@ + self.log.critical(_("\t$ python -m notebook.auth password")) + sys.exit(1) + ++ # Socket options validation. ++ if self.sock: ++ if self.port != DEFAULT_NOTEBOOK_PORT: ++ self.log.critical( ++ _('Options --port and --sock are mutually exclusive. Aborting.'), ++ ) ++ sys.exit(1) ++ ++ if self.open_browser: ++ # If we're bound to a UNIX socket, we can't reliably connect from a browser. ++ self.log.critical( ++ _('Options --open-browser and --sock are mutually exclusive. Aborting.'), ++ ) ++ sys.exit(1) ++ ++ if sys.platform.startswith('win'): ++ self.log.critical( ++ _('Option --sock is not supported on Windows, but got value of %s. Aborting.' % self.sock), ++ ) ++ sys.exit(1) ++ + self.web_app = NotebookWebApplication( + self, self.kernel_manager, self.contents_manager, + self.session_manager, self.kernel_spec_manager, +@@ -1401,6 +1477,32 @@ + max_body_size=self.max_body_size, + max_buffer_size=self.max_buffer_size) + ++ success = self._bind_http_server() ++ if not success: ++ self.log.critical(_('ERROR: the notebook server could not be started because ' ++ 'no available port could be found.')) ++ self.exit(1) ++ ++ def _bind_http_server(self): ++ return self._bind_http_server_unix() if self.sock else self._bind_http_server_tcp() ++ ++ def _bind_http_server_unix(self): ++ try: ++ sock = bind_unix_socket(self.sock, mode=int(self.sock_umask.encode(), 8)) ++ self.http_server.add_socket(sock) ++ except socket.error as e: ++ if e.errno == errno.EADDRINUSE: ++ self.log.info(_('The socket %s is already in use.') % self.sock) ++ return False ++ elif e.errno in (errno.EACCES, getattr(errno, 'WSAEACCES', errno.EACCES)): ++ self.log.warning(_("Permission to listen on sock %s denied") % self.sock) ++ return False ++ else: ++ raise ++ else: ++ return True ++ ++ def _bind_http_server_tcp(self): + success = None + for port in random_ports(self.port, self.port_retries+1): + try: +@@ -1418,10 +1520,11 @@ + self.port = port + success = True + break +- if not success: +- self.log.critical(_('ERROR: the notebook server could not be started because ' +- 'no available port could be found.')) +- self.exit(1) ++ return success ++ ++ def _concat_token(self, url): ++ token = self.token if self._token_generated else '...' ++ return url_concat(url, {'token': token}) + + @property + def display_url(self): +@@ -1429,26 +1532,33 @@ + url = self.custom_display_url + if not url.endswith('/'): + url += '/' ++ elif self.sock: ++ url = self._unix_sock_url() + else: + if self.ip in ('', '0.0.0.0'): + ip = "(%s or 127.0.0.1)" % socket.gethostname() + else: + ip = self.ip +- url = self._url(ip) +- if self.token: +- # Don't log full token if it came from config +- token = self.token if self._token_generated else '...' +- url = url_concat(url, {'token': token}) ++ url = self._tcp_url(ip) ++ if self.token and not self.sock: ++ url = self._concat_token(url) ++ url += '\n or %s' % self._concat_token(self._tcp_url('127.0.0.1')) + return url + + @property + def connection_url(self): +- ip = self.ip if self.ip else 'localhost' +- return self._url(ip) ++ if self.sock: ++ return self._unix_sock_url() ++ else: ++ ip = self.ip if self.ip else 'localhost' ++ return self._tcp_url(ip) + +- def _url(self, ip): ++ def _unix_sock_url(self, token=None): ++ return '%s%s' % (urlencode_unix_socket(self.sock), self.base_url) ++ ++ def _tcp_url(self, ip, port=None): + proto = 'https' if self.certfile else 'http' +- return "%s://%s:%i%s" % (proto, ip, self.port, self.base_url) ++ return "%s://%s:%i%s" % (proto, ip, port or self.port, self.base_url) + + def init_terminals(self): + if not self.terminals_enabled: +@@ -1660,6 +1770,7 @@ + return {'url': self.connection_url, + 'hostname': self.ip if self.ip else 'localhost', + 'port': self.port, ++ 'sock': self.sock, + 'secure': bool(self.certfile), + 'base_url': self.base_url, + 'token': self.token, +@@ -1780,19 +1891,31 @@ + self.write_server_info_file() + self.write_browser_open_file() + +- if self.open_browser or self.file_to_run: ++ if (self.open_browser or self.file_to_run) and not self.sock: + self.launch_browser() + + if self.token and self._token_generated: + # log full URL with generated token, so there's a copy/pasteable link + # with auth info. +- self.log.critical('\n'.join([ +- '\n', +- 'To access the notebook, open this file in a browser:', +- ' %s' % urljoin('file:', pathname2url(self.browser_open_file)), +- 'Or copy and paste one of these URLs:', +- ' %s' % self.display_url, +- ])) ++ if self.sock: ++ self.log.critical('\n'.join([ ++ '\n', ++ 'Notebook is listening on %s' % self.display_url, ++ '', ++ ( ++ 'UNIX sockets are not browser-connectable, but you can tunnel to ' ++ 'the instance via e.g.`ssh -L 8888:%s -N user@this_host` and then ' ++ 'opening e.g. %s in a browser.' ++ ) % (self.sock, self._concat_token(self._tcp_url('localhost', 8888))) ++ ])) ++ else: ++ self.log.critical('\n'.join([ ++ '\n', ++ 'To access the notebook, open this file in a browser:', ++ ' %s' % urljoin('file:', pathname2url(self.browser_open_file)), ++ 'Or copy and paste one of these URLs:', ++ ' %s' % self.display_url, ++ ])) + + self.io_loop = ioloop.IOLoop.current() + if sys.platform.startswith('win'): +diff -Naur notebook-5.7.4/notebook/tests/launchnotebook.py notebook-5.7.4.patched/notebook/tests/launchnotebook.py +--- notebook-5.7.4/notebook/tests/launchnotebook.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/tests/launchnotebook.py 2019-11-18 12:22:25.931074384 +0100 +@@ -19,12 +19,13 @@ + from mock import patch #py2 + + import requests ++import requests_unixsocket + from tornado.ioloop import IOLoop + import zmq + + import jupyter_core.paths + from traitlets.config import Config +-from ..notebookapp import NotebookApp ++from ..notebookapp import NotebookApp, urlencode_unix_socket + from ..utils import url_path_join + from ipython_genutils.tempdir import TemporaryDirectory + +@@ -55,7 +56,7 @@ + url = cls.base_url() + 'api/contents' + for _ in range(int(MAX_WAITTIME/POLL_INTERVAL)): + try: +- requests.get(url) ++ cls.fetch_url(url) + except Exception as e: + if not cls.notebook_thread.is_alive(): + raise RuntimeError("The notebook server failed to start") +@@ -79,6 +80,10 @@ + headers['Authorization'] = 'token %s' % cls.token + return headers + ++ @staticmethod ++ def fetch_url(url): ++ return requests.get(url) ++ + @classmethod + def request(cls, verb, path, **kwargs): + """Send a request to my server +@@ -93,6 +98,10 @@ + return response + + @classmethod ++ def get_bind_args(cls): ++ return dict(port=cls.port) ++ ++ @classmethod + def setup_class(cls): + cls.tmp_dir = TemporaryDirectory() + def tmp(*parts): +@@ -103,7 +112,7 @@ + if e.errno != errno.EEXIST: + raise + return path +- ++ + cls.home_dir = tmp('home') + data_dir = cls.data_dir = tmp('data') + config_dir = cls.config_dir = tmp('config') +@@ -138,8 +147,8 @@ + if 'asyncio' in sys.modules: + import asyncio + asyncio.set_event_loop(asyncio.new_event_loop()) ++ bind_args = cls.get_bind_args() + app = cls.notebook = NotebookApp( +- port=cls.port, + port_retries=0, + open_browser=False, + config_dir=cls.config_dir, +@@ -150,6 +159,7 @@ + config=config, + allow_root=True, + token=cls.token, ++ **bind_args + ) + # don't register signal handler during tests + app.init_signal = lambda : None +@@ -197,6 +207,25 @@ + return 'http://localhost:%i%s' % (cls.port, cls.url_prefix) + + ++class UNIXSocketNotebookTestBase(NotebookTestBase): ++ # Rely on `/tmp` to avoid any Linux socket length max buffer ++ # issues. Key on PID for process-wise concurrency. ++ sock = '/tmp/.notebook.%i.sock' % os.getpid() ++ ++ @classmethod ++ def get_bind_args(cls): ++ return dict(sock=cls.sock) ++ ++ @classmethod ++ def base_url(cls): ++ return '%s%s' % (urlencode_unix_socket(cls.sock), cls.url_prefix) ++ ++ @staticmethod ++ def fetch_url(url): ++ with requests_unixsocket.monkeypatch(): ++ return requests.get(url) ++ ++ + @contextmanager + def assert_http_error(status, msg=None): + try: +diff -Naur notebook-5.7.4/notebook/tests/test_notebookapp_integration.py notebook-5.7.4.patched/notebook/tests/test_notebookapp_integration.py +--- notebook-5.7.4/notebook/tests/test_notebookapp_integration.py 1970-01-01 01:00:00.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/tests/test_notebookapp_integration.py 2019-11-18 12:16:58.319065025 +0100 +@@ -0,0 +1,39 @@ ++import os ++import stat ++import subprocess ++import time ++ ++from ipython_genutils.testing.decorators import skip_win32 ++ ++from .launchnotebook import UNIXSocketNotebookTestBase ++from ..utils import urlencode_unix_socket, urlencode_unix_socket_path ++ ++ ++@skip_win32 ++def test_shutdown_sock_server_integration(): ++ sock = UNIXSocketNotebookTestBase.sock ++ url = urlencode_unix_socket(sock) ++ encoded_sock_path = urlencode_unix_socket_path(sock) ++ ++ p = subprocess.Popen( ++ ['jupyter', 'notebook', '--no-browser', '--sock=%s' % sock], ++ stdout=subprocess.PIPE, stderr=subprocess.PIPE ++ ) ++ ++ for line in iter(p.stderr.readline, b''): ++ if url.encode() in line: ++ complete = True ++ break ++ ++ assert complete, 'did not find socket URL in stdout when launching notebook' ++ ++ assert encoded_sock_path.encode() in subprocess.check_output(['jupyter', 'notebook', 'list']) ++ ++ # Ensure default umask is properly applied. ++ assert stat.S_IMODE(os.lstat(sock).st_mode) == 0o600 ++ ++ subprocess.check_output(['jupyter', 'notebook', 'stop', sock]) ++ ++ assert encoded_sock_path.encode() not in subprocess.check_output(['jupyter', 'notebook', 'list']) ++ ++ p.wait() +diff -Naur notebook-5.7.4/notebook/tests/test_notebookapp.py notebook-5.7.4.patched/notebook/tests/test_notebookapp.py +--- notebook-5.7.4/notebook/tests/test_notebookapp.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/tests/test_notebookapp.py 2019-11-18 12:16:58.319065025 +0100 +@@ -25,7 +25,7 @@ + from notebook.auth.security import passwd_check + NotebookApp = notebookapp.NotebookApp + +-from .launchnotebook import NotebookTestBase ++from .launchnotebook import NotebookTestBase, UNIXSocketNotebookTestBase + + + def test_help_output(): +@@ -192,3 +192,15 @@ + servers = list(notebookapp.list_running_servers()) + assert len(servers) >= 1 + assert self.port in {info['port'] for info in servers} ++ ++ ++# UNIX sockets aren't available on Windows. ++if not sys.platform.startswith('win'): ++ class NotebookUnixSocketTests(UNIXSocketNotebookTestBase): ++ def test_run(self): ++ self.fetch_url(self.base_url() + 'api/contents') ++ ++ def test_list_running_sock_servers(self): ++ servers = list(notebookapp.list_running_servers()) ++ assert len(servers) >= 1 ++ assert self.sock in {info['sock'] for info in servers} +diff -Naur notebook-5.7.4/notebook/utils.py notebook-5.7.4.patched/notebook/utils.py +--- notebook-5.7.4/notebook/utils.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/notebook/utils.py 2019-11-18 12:23:05.231075507 +0100 +@@ -306,3 +306,18 @@ + check_pid = _check_pid_win32 + else: + check_pid = _check_pid_posix ++ ++def urlencode_unix_socket_path(socket_path): ++ """Encodes a UNIX socket path string from a socket path for the `http+unix` URI form.""" ++ return socket_path.replace('/', '%2F') ++ ++ ++def urldecode_unix_socket_path(socket_path): ++ """Decodes a UNIX sock path string from an encoded sock path for the `http+unix` URI form.""" ++ return socket_path.replace('%2F', '/') ++ ++ ++def urlencode_unix_socket(socket_path): ++ """Encodes a UNIX socket URL from a socket path for the `http+unix` URI form.""" ++ return 'http+unix://%s' % urlencode_unix_socket_path(socket_path) ++ +diff -Naur notebook-5.7.4/setup.py notebook-5.7.4.patched/setup.py +--- notebook-5.7.4/setup.py 2018-12-17 11:01:51.000000000 +0100 ++++ notebook-5.7.4.patched/setup.py 2019-11-18 12:23:33.851076325 +0100 +@@ -98,7 +98,8 @@ + ':python_version == "2.7"': ['ipaddress'], + 'test:python_version == "2.7"': ['mock'], + 'test': ['nose', 'coverage', 'requests', 'nose_warnings_filters', +- 'nbval', 'nose-exclude', 'selenium'], ++ 'nbval', 'nose-exclude', 'selenium', ++ 'requests-unixsocket'], + 'test:sys_platform == "win32"': ['nose-exclude'], + }, + entry_points = { diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 232841ccb1..4263a33c6b 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -7811,7 +7811,8 @@ convert an @code{.ipynb} notebook file into various static formats including: (uri (pypi-uri "notebook" version)) (sha256 (base32 - "0jm7324mbxljmn9hgapj66q7swyz5ai92blmr0jpcy0h80x6f26r")))) + "0jm7324mbxljmn9hgapj66q7swyz5ai92blmr0jpcy0h80x6f26r")) + (patches (search-patches "jupyter-unix-domain-sockets-4835-5.7.4.patch")))) (build-system python-build-system) (arguments `(#:phases @@ -7834,7 +7835,8 @@ convert an @code{.ipynb} notebook file into various static formats including: ("python-nbconvert" ,python-nbconvert) ("python-prometheus-client" ,python-prometheus-client) ("python-send2trash" ,python-send2trash) - ("python-terminado" ,python-terminado))) + ("python-terminado" ,python-terminado) + ("python-requests-unixsocket" ,python-requests-unixsocket))) (native-inputs `(("python-nose" ,python-nose) ("python-sphinx" ,python-sphinx) @@ -17711,3 +17713,32 @@ applications") based on the Jupyter Notebook and Architecture.") (license license:bsd-3))) +(define-public python-requests-unixsocket + (package + (name "python-requests-unixsocket") + (version "0.2.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "requests-unixsocket" version)) + (sha256 + (base32 + "1sn12y4fw1qki5gxy9wg45gmdrxhrndwfndfjxhpiky3mwh1lp4y")))) + (build-system python-build-system) + (native-inputs + ;; pbr is required for setup only + `(("python-pbr" ,python-pbr))) + (propagated-inputs + `(("python-requests" ,python-requests) + ("python-urllib3" ,python-urllib3))) + (arguments + ;; tests depend on very specific package version, which are not available in guix + '(#:tests? #f)) + (home-page + "https://github.com/msabramo/requests-unixsocket") + (synopsis + "Use requests to talk HTTP via a UNIX domain socket") + (description + "Use requests to talk HTTP via a UNIX domain socket") + (license license:asl2.0))) + -- 2.20.1 --sdtB3X0nJg68CQEu--