From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id mAavB6RuiGDTCQEAgWs5BA (envelope-from ) for ; Tue, 27 Apr 2021 22:05:56 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id wZgzA6RuiGAsDgAAbx9fmQ (envelope-from ) for ; Tue, 27 Apr 2021 20:05:56 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6AF7813CA6 for ; Tue, 27 Apr 2021 22:05:55 +0200 (CEST) Received: from localhost ([::1]:33494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lbTxx-000295-QH for larch@yhetil.org; Tue, 27 Apr 2021 16:05:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51602) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lbTvl-0000qP-JT for help-guix@gnu.org; Tue, 27 Apr 2021 16:03:37 -0400 Received: from sender4-op-o10.zoho.com ([136.143.188.10]:17070) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lbTvj-0004A5-9Y for help-guix@gnu.org; Tue, 27 Apr 2021 16:03:37 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1619553811; cv=none; d=zohomail.com; s=zohoarc; b=CabX+low6XjzH7ybg/nHAp+lnI3KLGChSPIIOrft4a2hNYS+/w6tuXBBi+o/VYzb9YbO1jBHt838QQoWJ0XOR5efgpzY5A2i3KA1pfpOTSU4saaQX8vESNL6pAfX+XrtzMIrpmJjYJ0DK3ZIc238B9Y35vUo63XvLIRZ/3Bm+Zg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1619553811; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=rbG6rRQbrQmNfQI8w4L4TeGkDynTv55TQQqEW67ezTY=; b=c2A9tx+X2agiTjlFw0fgPsGuhpXkesrEjLwNKYfInnZhLs5bNYA6xOBAZEXfe61n9DWCLLlwpCmgrVwazUOgZJf6xlaXz52VO4T8bf9R1ztrYINflx7XU7Is7UqpaC5WgXaEVYyQdPrRvUe91piF8OKEdTTi9O1LK0q3nUXWlj8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=rdklein.fr; spf=pass smtp.mailfrom=edou@rdklein.fr; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619553811; s=zoho; d=rdklein.fr; i=edou@rdklein.fr; h=References:From:To:Cc:Subject:Message-ID:In-reply-to:Date:MIME-Version:Content-Type; bh=rbG6rRQbrQmNfQI8w4L4TeGkDynTv55TQQqEW67ezTY=; b=eot98xwVNyutxBzeZ6ChvzJd0dzJvZzT0iera23oU6mppOSApO1hawjP2LitljXA kc7lIw14AYFp6LBx6XRWpOlg7HuB2Clog586gbcPMOn41i916gk+TiY5+sowAezxObJ +I8hal1s4Cdmv4aZlvHCO8QYP9GP0cObFCR3tFYE= Received: from Rasoir (lfbn-idf3-1-808-29.w90-3.abo.wanadoo.fr [90.3.133.29]) by mx.zohomail.com with SMTPS id 1619553807504877.0027882331837; Tue, 27 Apr 2021 13:03:27 -0700 (PDT) References: <87r1jgsjme.fsf@rdklein.fr> <75a95dd5315d1633452694f824b42141989d9c03.camel@telenet.be> User-agent: mu4e 1.4.15; emacs 27.1 From: Edouard Klein To: Maxime Devos Subject: Re: Environment of a shepherd service Message-ID: <877dko5mgc.fsf@rdklein.fr> In-reply-to: <75a95dd5315d1633452694f824b42141989d9c03.camel@telenet.be> Date: Tue, 27 Apr 2021 22:03:08 +0200 MIME-Version: 1.0 Content-Type: text/plain X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.10; envelope-from=edou@rdklein.fr; helo=sender4-op-o10.zoho.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix@gnu.org Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619553955; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=rbG6rRQbrQmNfQI8w4L4TeGkDynTv55TQQqEW67ezTY=; b=hjyM4m1wy/ef409S8HjPF+9HqxnB8JAL9wbXSi9Jw9RSnsKwVOK0ERQmjv1iDc2Q6+Kj3t 4TpMJ+fcRrqDUqaMGxlekzdKC3EZKgkX3sbmAZnC5N3byUaE9TvKrC2/CwTRipbuVfneqU UBP+uHjpTuxacbgUX0KmEq+Nz/LVT5JPFyF5UXBFkRfBldi48r6V/IS7B1lnug3vs/pSol VsybN4XHpUXHAEx+VA3A6ootdvT6GhIwQEujaXtRWQyMnIxS/YZJsioK3niBu3NFyHb0fr w8WGme29tk5wrtnEp79wCun/vaZxqFuCjclaVwQkmuxouh2r+JJ7dLQyqwhc3g== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1619553955; a=rsa-sha256; cv=pass; b=GFyi6zPS1hFZX0jvBc7nXsKnz0vSkhFknt3cC583qUzaNHlGJX38+cXb7PsosH2yKO9UrA 1fzZT+VC7xvhAQdHiLVX4bSfERaiY3dmMdEWEq31SD0Av27725JVsqlNmRh8fFSuKcpe9c C5sbYTC63suAPk84fRv814LLW1fQZbOzUJr5CoVBHgqo7IX7cwpZNJNLYVyBLzRH3npElR 9GAufVlT6ukANVFjSOOE4iiRwGm42Kdu/l3/FMcBdORvYMFl9i1Nb2IC3Hti21GaIVsrIU +3+wSnY+96hLeMtrp2/xVxBc6P8WLf26nUi3s9V2+Fb3ryO2d4HNYKHshJFiHQ== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=none ("invalid DKIM record") header.d=rdklein.fr header.s=zoho header.b=eot98xwV; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Spam-Score: -3.45 Authentication-Results: aspmx1.migadu.com; dkim=none ("invalid DKIM record") header.d=rdklein.fr header.s=zoho header.b=eot98xwV; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Queue-Id: 6AF7813CA6 X-Spam-Score: -3.45 X-Migadu-Scanner: scn0.migadu.com X-TUID: H88EbXG73tzO Thank you Maxime for your answer :) Maxime Devos writes: > edk@beaver-labs.com schreef op zo 11-04-2021 om 21:31 [+0200]: >> Dear fellow Guixers, > >> [...] >> But, when I try to run it with shepherd, it fails because it can't find >> flask (a dependency of the software, which I've put as a >> propagated-input, and is indeed installed in the container). > > Propagated inputs can be inconvenient at times. I would advise > looking where requisomatic is referring to flask, and replacing > flask --> (string-append (assoc-ref inputs "flask") "/bin/flask") > using substitute*. OK, I understand this part :) The executable is gunicorn, so I'll just use its full path in the script that launches the service. My previous solution was to source the profile beforehand: https://gitlab.com/edouardklein/requisomatic/-/blob/baf3fe51ad8bbabbcbc467dff92ec02c43e6daf1/guix.scm#L200 I do agree that a rich profile feels dirty and I too don't like propagated inputs too much, but with Python I don't understand what the best way to do it is ? For example, in order to package a Python application in a way that does not break the host system, I had to put all the search-paths in a wrapper script and change all propagated inputs as just inputs. I find the resulting code quite ugly, and I don't think I'm doing things right: https://gitlab.com/edouardklein/gendscraper/-/blob/980f2597cc36bc79a9be7af5814e0fcf2313b677/gendscraper.scm#L1176 I wouldn't like to have to this for every service. If propagated inputs are inconvenient, and if services run in an environment where the search-paths of the installed packages are not available, how can we easily tell the software where to find its dynamically-loaded parts ? > >> But, when I try to run it with shepherd, it fails because it can't find >> flask (a dependency of the software, which I've put as a >> propagated-input, and is indeed installed in the container). >> [...] > > Some advice (warning: I'm not familiar with gunicorn or requisomatic > at all). I used flask and gunicorn in this project, but I'm not familiar with them either ;) > >> >> -----extract from my operating-system declaration file------- >> (define requisomatic-shepherd-service >> ([...] (shepherd-service >> [...] > (documentation "Run the requisomatic server") >> (start #~((make-forkexec-constructor >> ;; (append >> ;; (if db-file >> ;; `("env" >> ;; ,(string-append "REQUISOMATIC_DB_FILE=" db-file)) >> ;; '()) >> '("gunicorn" "requisomatic:app") > > Normally, services refer by absolute path to the binary to run, and not rely > on the PATH (the latter would require polluting the system profile). > Idiomatically, one would write > > '(#$(file-append gunicorn "/bin/gunicorn") > #$(file-append requisomatic) "/wherever/the/binary/is") > I did not know about file-append even though it is in the manual. Thanks :) >> #:directory (string-append #$requisomatic "/bin/requisomatic/") > > Why are you changing the working directory to > (string-append #$requisomatic "/bin/requisomatic/"), because . is by default where flask will look for the code it needs. >and why is "/bin/requisomatic" a > directory and not an executable? Because flask/jinja/etc. are normative about where they want stuff to be and I put everything in a single dir. As most of the assets is code (some are html templates) I put it in bin/... but I could put them in /lib of wherever. This is not the cleanest, but I really wanted to get a guix operating system declaration to work before I did things cleanly. > Is that a gunicorn thing? More like flask, but yeah, it's a framework thing. > >> Why is the PYTHONPATH (and the other env vars, for that matter) not >> propagated from the package to the shepherd service by default ? > > How is the shepherd service supposed to automagically know which packages > to include in the environment variables? > I realised afterwards that by extending the profile-service-type, services can say what packages they need (I currently add them to the operating-system definition). I would expect the shepherd command to run in an environment where the search-paths of those package is set. I have no idea if this is complicated to do, but as a service writer I would enjoy it very much. >> And how can I make it so ? > > Use the #:environment-variables option, see e.g. > bitlbee-shepherd-service I see the concept, but the list is hardcoded, and for even moderately complex Python application (or any other dynamic language) the list of env variables to set will become huge. > Or create a wrapper. See e.g. wrapped-dbus-service. This looks like a clean version of what I did for gendscraper, but the problem remains of which variable to set. This information exists in each individual package. Some will need additions to e.g. PYTHONPATH, others to LD_LIBRARY_PATH, etc. We need to walk the entire dependency DAG somehow. > >> Follow up question, can shepherd services be specified to run in a >> specific profile ? > > IIUC, currently shepherd services aren't run in *any* profile at all. > It would be useful to have a function manifest->environment-gexp > though. Or package->search-paths ? > >> So that I can have two services with incompatible >> dependencies running at the same time in the same operating-system ? > Yes, it with "dependencies" you mean packages, and not other services. > I meant packages :) Thank you very much for your help, it seems you know services inside out. I can now pinpoint way better what it is that I don't understand. I hope I was more clear in expressing it. Cheers ! Edouard. > Greetings, > Maxime.