Environment of a shepherd service

Environment of a shepherd service

[edk]

Dear fellow Guixers,

I'm trying to create an operating system declaration, so that I can run
a piece of software of mine in a container with =guix system container=.

I wrote a package for the software. The package works: the tests pass
and when the package is installed I can run the software.

I wrote a shepherd service for the software (it's called requisomatic).
I copied the relevant part at the end of the email.

When I run the container script created by =guix system container=, and
get a shell in the container, I can run the software (I added the software's
package to the globally installed packages in the operating-system definition).

But, when I try to run it with shepherd, it fails because it can't find
flask (a dependency of the software, which I've put as a
propagated-input, and is indeed installed in the container).

I replaced the software invocation in the shepherd service with just
"env", and saw that the whole env in the service is:

PATH=/run/current-system/profile/bin

whereas in the shell I get when I connect to the container, the env
contains many other variables, including a correctly set PYTHONPATH,
which allows the finding of flask.

So I now know why my software is not starting, but my question is:

Why is the PYTHONPATH (and the other env vars, for that matter) not
propagated from the package to the shepherd service by default ? And how
can I make it so ? I would have expected the shepherd service to run
with the global profile active.

Follow up question, can shepherd services be specified to run in a
specific profile ? So that I can have two services with incompatible
dependencies running at the same time in the same operating-system ?

Thanks in advance,

Cheers,

Edouard.



-----extract from my operating-system declaration file-------
(define requisomatic-shepherd-service
  (match-lambda
    (($ <requisomatic-configuration> user group db-file)
     (list (shepherd-service
            (provision '(requisomatic))
            (requirement '(user-processes networking))
            (documentation "Run the requisomatic server")
            (start #~((make-forkexec-constructor
                    ;;   (append
                    ;;    (if db-file
                    ;;      `("env"
                    ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
                    ;;      '())
                                        '("gunicorn" "requisomatic:app")
                   ;;   '("env")
                       ;;)
                       #:directory (string-append #$requisomatic "/bin/requisomatic/")
                       #:log-file "/var/log/requisomatic.log")))
            (stop #~(make-kill-destructor)))))))

Re: Environment of a shepherd service

[Edouard Klein]

Hi,

So I looked at the source and I understand that there's no way around
having only PATH=/run/current-system/profile/bin as the sole environment
of a service (which makes me wonder how anyone is running any service in
GuixSD, don't you need any env variables ?).

I tried to define a trivial package that would use wrap-program to
create a script that would set the environment variables to all the
search-paths of my requisomatic package, but I don't know how to access
those !

In the code that is executed by the daemon, all references to the
package are lost, it is not in the same strata as the package.

I can get the store path to the package but that does not help me.

I really could use some guidance here.

Cheers,

Edouard.
edk@beaver-labs.com writes:

> Dear fellow Guixers,
>
> I'm trying to create an operating system declaration, so that I can run
> a piece of software of mine in a container with =guix system container=.
>
> I wrote a package for the software. The package works: the tests pass
> and when the package is installed I can run the software.
>
> I wrote a shepherd service for the software (it's called requisomatic).
> I copied the relevant part at the end of the email.
>
> When I run the container script created by =guix system container=, and
> get a shell in the container, I can run the software (I added the software's
> package to the globally installed packages in the operating-system definition).
>
> But, when I try to run it with shepherd, it fails because it can't find
> flask (a dependency of the software, which I've put as a
> propagated-input, and is indeed installed in the container).
>
> I replaced the software invocation in the shepherd service with just
> "env", and saw that the whole env in the service is:
>
> PATH=/run/current-system/profile/bin
>
> whereas in the shell I get when I connect to the container, the env
> contains many other variables, including a correctly set PYTHONPATH,
> which allows the finding of flask.
>
> So I now know why my software is not starting, but my question is:
>
> Why is the PYTHONPATH (and the other env vars, for that matter) not
> propagated from the package to the shepherd service by default ? And how
> can I make it so ? I would have expected the shepherd service to run
> with the global profile active.
>
> Follow up question, can shepherd services be specified to run in a
> specific profile ? So that I can have two services with incompatible
> dependencies running at the same time in the same operating-system ?
>
> Thanks in advance,
>
> Cheers,
>
> Edouard.
>
>
>
> -----extract from my operating-system declaration file-------
> (define requisomatic-shepherd-service
>   (match-lambda
>     (($ <requisomatic-configuration> user group db-file)
>      (list (shepherd-service
>             (provision '(requisomatic))
>             (requirement '(user-processes networking))
>             (documentation "Run the requisomatic server")
>             (start #~((make-forkexec-constructor
>                     ;;   (append
>                     ;;    (if db-file
>                     ;;      `("env"
>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
>                     ;;      '())
>                                         '("gunicorn" "requisomatic:app")
>                    ;;   '("env")
>                        ;;)
>                        #:directory (string-append #$requisomatic "/bin/requisomatic/")
>                        #:log-file "/var/log/requisomatic.log")))
>             (stop #~(make-kill-destructor)))))))

Re: Environment of a shepherd service

[Jonathan McHugh]

Did you read this blog post?:
https://guix.gnu.org/en/blog/2020/gnu-shepherd-user-services/

====== START ========
The GNU Shepherd manual suggests putting all the services inside a monolithic init.scm file, located by default at $XDG_CONFIG_DIR/shepherd/init.scm. While this does make it easy to keep everything in one place, it does create one glaring issue: any changes to the file mean that all the services need to be stopped and restarted in order for any changes to take place.

Luckily there's a nice function called scandir hiding in ice-9 ftw which returns a list of all files in a specified directory (with options for narrowing down the list or sorting it). This means that our init.scm can contain a minimum of code and all actual services can be loaded from individual files.

First the minimal init.scm:

(use-modules (shepherd service)
             ((ice-9 ftw) #:select (scandir)))

;; Load all the files in the directory 'init.d' with a suffix '.scm'.
(for-each
  (lambda (file)
    (load (string-append "init.d/" file)))
  (scandir (string-append (dirname (current-filename)) "/init.d")
           (lambda (file)
             (string-suffix? ".scm" file))))

;; Send shepherd into the background
(action 'shepherd 'daemonize)

Let's take a sample service for running syncthing, as defined in $XDG_CONFIG_DIR/shepherd/init.d/syncthing.scm:

(define syncthing
  (make <service>
    #:provides '(syncthing)
    #:docstring "Run `syncthing' without calling the browser"
    #:start (make-forkexec-constructor
              '("syncthing" "-no-browser")
              #:log-file (string-append (getenv "HOME")
                                        "/log/syncthing.log"))
    #:stop (make-kill-destructor)
    #:respawn? #t))
(register-services syncthing)

(start syncthing)

As with any other shepherd service it is defined and registered, and in
this case it will start automatically. When the file is loaded by
shepherd after being discovered by scandir everything works exactly as
though the service definition were located directly inside the init.scm.

======== END =======


HTH


Edouard Klein <edou@rdklein.fr> writes:

> Hi,
>
> So I looked at the source and I understand that there's no way around
> having only PATH=/run/current-system/profile/bin as the sole environment
> of a service (which makes me wonder how anyone is running any service in
> GuixSD, don't you need any env variables ?).
>
> I tried to define a trivial package that would use wrap-program to
> create a script that would set the environment variables to all the
> search-paths of my requisomatic package, but I don't know how to access
> those !
>
> In the code that is executed by the daemon, all references to the
> package are lost, it is not in the same strata as the package.
>
> I can get the store path to the package but that does not help me.
>
> I really could use some guidance here.
>
> Cheers,
>
> Edouard.
> edk@beaver-labs.com writes:
>
>> Dear fellow Guixers,
>>
>> I'm trying to create an operating system declaration, so that I can run
>> a piece of software of mine in a container with =guix system container=.
>>
>> I wrote a package for the software. The package works: the tests pass
>> and when the package is installed I can run the software.
>>
>> I wrote a shepherd service for the software (it's called requisomatic).
>> I copied the relevant part at the end of the email.
>>
>> When I run the container script created by =guix system container=, and
>> get a shell in the container, I can run the software (I added the software's
>> package to the globally installed packages in the operating-system definition).
>>
>> But, when I try to run it with shepherd, it fails because it can't find
>> flask (a dependency of the software, which I've put as a
>> propagated-input, and is indeed installed in the container).
>>
>> I replaced the software invocation in the shepherd service with just
>> "env", and saw that the whole env in the service is:
>>
>> PATH=/run/current-system/profile/bin
>>
>> whereas in the shell I get when I connect to the container, the env
>> contains many other variables, including a correctly set PYTHONPATH,
>> which allows the finding of flask.
>>
>> So I now know why my software is not starting, but my question is:
>>
>> Why is the PYTHONPATH (and the other env vars, for that matter) not
>> propagated from the package to the shepherd service by default ? And how
>> can I make it so ? I would have expected the shepherd service to run
>> with the global profile active.
>>
>> Follow up question, can shepherd services be specified to run in a
>> specific profile ? So that I can have two services with incompatible
>> dependencies running at the same time in the same operating-system ?
>>
>> Thanks in advance,
>>
>> Cheers,
>>
>> Edouard.
>>
>>
>>
>> -----extract from my operating-system declaration file-------
>> (define requisomatic-shepherd-service
>>   (match-lambda
>>     (($ <requisomatic-configuration> user group db-file)
>>      (list (shepherd-service
>>             (provision '(requisomatic))
>>             (requirement '(user-processes networking))
>>             (documentation "Run the requisomatic server")
>>             (start #~((make-forkexec-constructor
>>                     ;;   (append
>>                     ;;    (if db-file
>>                     ;;      `("env"
>>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
>>                     ;;      '())
>>                                         '("gunicorn" "requisomatic:app")
>>                    ;;   '("env")
>>                        ;;)
>>                        #:directory (string-append #$requisomatic "/bin/requisomatic/")
>>                        #:log-file "/var/log/requisomatic.log")))
>>             (stop #~(make-kill-destructor)))))))


-- 
Jonathan McHugh
indieterminacy@libre.brussels

Re: Environment of a shepherd service

[Edouard Klein]

Hi !

I did, thanks :) But this post talks only about shepherd, not its
integration with guix. It is indeed a very good resource for creating
shepherd services, and I was able thanks to it to write a shepherd
service.

What I would like to do is create a shepherd service, using some
facilities offered by the "operating-system declaration" concept of
guix.

In this instance I have trouble setting the env of the service to the
search paths of the installed packages.

The final goal is to orchestrate multiple services (typically multiple
web services and an instance of nginx to reverse-proxy them all) in one
operating-system declaration.

Here are the resources I have found on this topic


  - https://www.gnu.org/software/shepherd/manual/shepherd.html#Services
  - https://guix.gnu.org/en/blog/2020/gnu-shepherd-user-services/
  - https://archive.fosdem.org/2017/schedule/event/composingsystemservicesinguixsd/attachments/slides/1794/export/events/attachments/composingsystemservicesinguixsd/slides/1794/guix_service_composition.pdf
  - https://www.mndet.net/2016/05/04/guixsd-system-service.html
  - https://guix.gnu.org/en/manual/en/guix.html#Defining-Services

The documentation is scarce (not an attack, I understand writing docs is
harder than writing code), and I have trouble getting used to the source
because I'm unable to get a guile repl at the level at which I'm trying
to run code, so I have to make edits, build the container, try, repeat.
If there is an error I have no stack trace or debug information.

Cheers,

Edouard.

Jonathan McHugh writes:

> Did you read this blog post?:
> https://guix.gnu.org/en/blog/2020/gnu-shepherd-user-services/
>
> ====== START ========
> The GNU Shepherd manual suggests putting all the services inside a monolithic init.scm file, located by default at $XDG_CONFIG_DIR/shepherd/init.scm. While this does make it easy to keep everything in one place, it does create one glaring issue: any changes to the file mean that all the services need to be stopped and restarted in order for any changes to take place.
>
> Luckily there's a nice function called scandir hiding in ice-9 ftw which returns a list of all files in a specified directory (with options for narrowing down the list or sorting it). This means that our init.scm can contain a minimum of code and all actual services can be loaded from individual files.
>
> First the minimal init.scm:
>
> (use-modules (shepherd service)
>              ((ice-9 ftw) #:select (scandir)))
>
> ;; Load all the files in the directory 'init.d' with a suffix '.scm'.
> (for-each
>   (lambda (file)
>     (load (string-append "init.d/" file)))
>   (scandir (string-append (dirname (current-filename)) "/init.d")
>            (lambda (file)
>              (string-suffix? ".scm" file))))
>
> ;; Send shepherd into the background
> (action 'shepherd 'daemonize)
>
> Let's take a sample service for running syncthing, as defined in $XDG_CONFIG_DIR/shepherd/init.d/syncthing.scm:
>
> (define syncthing
>   (make <service>
>     #:provides '(syncthing)
>     #:docstring "Run `syncthing' without calling the browser"
>     #:start (make-forkexec-constructor
>               '("syncthing" "-no-browser")
>               #:log-file (string-append (getenv "HOME")
>                                         "/log/syncthing.log"))
>     #:stop (make-kill-destructor)
>     #:respawn? #t))
> (register-services syncthing)
>
> (start syncthing)
>
> As with any other shepherd service it is defined and registered, and in
> this case it will start automatically. When the file is loaded by
> shepherd after being discovered by scandir everything works exactly as
> though the service definition were located directly inside the init.scm.
>
> ======== END =======
>
>
> HTH
>
>
> Edouard Klein <edou@rdklein.fr> writes:
>
>> Hi,
>>
>> So I looked at the source and I understand that there's no way around
>> having only PATH=/run/current-system/profile/bin as the sole environment
>> of a service (which makes me wonder how anyone is running any service in
>> GuixSD, don't you need any env variables ?).
>>
>> I tried to define a trivial package that would use wrap-program to
>> create a script that would set the environment variables to all the
>> search-paths of my requisomatic package, but I don't know how to access
>> those !
>>
>> In the code that is executed by the daemon, all references to the
>> package are lost, it is not in the same strata as the package.
>>
>> I can get the store path to the package but that does not help me.
>>
>> I really could use some guidance here.
>>
>> Cheers,
>>
>> Edouard.
>> edk@beaver-labs.com writes:
>>
>>> Dear fellow Guixers,
>>>
>>> I'm trying to create an operating system declaration, so that I can run
>>> a piece of software of mine in a container with =guix system container=.
>>>
>>> I wrote a package for the software. The package works: the tests pass
>>> and when the package is installed I can run the software.
>>>
>>> I wrote a shepherd service for the software (it's called requisomatic).
>>> I copied the relevant part at the end of the email.
>>>
>>> When I run the container script created by =guix system container=, and
>>> get a shell in the container, I can run the software (I added the software's
>>> package to the globally installed packages in the operating-system definition).
>>>
>>> But, when I try to run it with shepherd, it fails because it can't find
>>> flask (a dependency of the software, which I've put as a
>>> propagated-input, and is indeed installed in the container).
>>>
>>> I replaced the software invocation in the shepherd service with just
>>> "env", and saw that the whole env in the service is:
>>>
>>> PATH=/run/current-system/profile/bin
>>>
>>> whereas in the shell I get when I connect to the container, the env
>>> contains many other variables, including a correctly set PYTHONPATH,
>>> which allows the finding of flask.
>>>
>>> So I now know why my software is not starting, but my question is:
>>>
>>> Why is the PYTHONPATH (and the other env vars, for that matter) not
>>> propagated from the package to the shepherd service by default ? And how
>>> can I make it so ? I would have expected the shepherd service to run
>>> with the global profile active.
>>>
>>> Follow up question, can shepherd services be specified to run in a
>>> specific profile ? So that I can have two services with incompatible
>>> dependencies running at the same time in the same operating-system ?
>>>
>>> Thanks in advance,
>>>
>>> Cheers,
>>>
>>> Edouard.
>>>
>>>
>>>
>>> -----extract from my operating-system declaration file-------
>>> (define requisomatic-shepherd-service
>>>   (match-lambda
>>>     (($ <requisomatic-configuration> user group db-file)
>>>      (list (shepherd-service
>>>             (provision '(requisomatic))
>>>             (requirement '(user-processes networking))
>>>             (documentation "Run the requisomatic server")
>>>             (start #~((make-forkexec-constructor
>>>                     ;;   (append
>>>                     ;;    (if db-file
>>>                     ;;      `("env"
>>>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
>>>                     ;;      '())
>>>                                         '("gunicorn" "requisomatic:app")
>>>                    ;;   '("env")
>>>                        ;;)
>>>                        #:directory (string-append #$requisomatic "/bin/requisomatic/")
>>>                        #:log-file "/var/log/requisomatic.log")))
>>>             (stop #~(make-kill-destructor)))))))

Re: Environment of a shepherd service

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2859 bytes --]

edk@beaver-labs.com schreef op zo 11-04-2021 om 21:31 [+0200]:
> Dear fellow Guixers,

> [...]
> But, when I try to run it with shepherd, it fails because it can't find
> flask (a dependency of the software, which I've put as a
> propagated-input, and is indeed installed in the container).

Propagated inputs can be inconvenient at times.  I would advise
looking where requisomatic is referring to flask, and replacing
flask --> (string-append (assoc-ref inputs "flask") "/bin/flask")
using substitute*.

> But, when I try to run it with shepherd, it fails because it can't find
> flask (a dependency of the software, which I've put as a
> propagated-input, and is indeed installed in the container).
> [...]

Some advice (warning: I'm not familiar with gunicorn or requisomatic at all).

> 
> -----extract from my operating-system declaration file-------
> (define requisomatic-shepherd-service
>   ([...] (shepherd-service
>             [...]
            (documentation "Run the requisomatic server")
>             (start #~((make-forkexec-constructor
>                     ;;   (append
>                     ;;    (if db-file
>                     ;;      `("env"
>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
>                     ;;      '())
>                                         '("gunicorn" "requisomatic:app")

Normally, services refer by absolute path to the binary to run, and not rely
on the PATH (the latter would require polluting the system profile).
Idiomatically, one would write

  '(#$(file-append gunicorn "/bin/gunicorn")
    #$(file-append requisomatic) "/wherever/the/binary/is")

>                        #:directory (string-append #$requisomatic "/bin/requisomatic/")

Why are you changing the working directory to
(string-append #$requisomatic "/bin/requisomatic/"), and why is "/bin/requisomatic" a
directory and not an executable?  Is that a gunicorn thing?

> Why is the PYTHONPATH (and the other env vars, for that matter) not
> propagated from the package to the shepherd service by default ?

How is the shepherd service supposed to automagically know which packages
to include in the environment variables?

> And how can I make it so ?

Use the #:environment-variables option, see e.g. bitlbee-shepherd-service
Or create a wrapper.  See e.g. wrapped-dbus-service.

> Follow up question, can shepherd services be specified to run in a
> specific profile ?

IIUC, currently shepherd services aren't run in *any* profile at all.
It would be useful to have a function manifest->environment-gexp though.

> So that I can have two services with incompatible
> dependencies running at the same time in the same operating-system ?
Yes, it with "dependencies" you mean packages, and not other services.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Environment of a shepherd service

[Edouard Klein]

Thank you Maxime for your answer :)

Maxime Devos writes:

> edk@beaver-labs.com schreef op zo 11-04-2021 om 21:31 [+0200]:
>> Dear fellow Guixers,
>
>> [...]
>> But, when I try to run it with shepherd, it fails because it can't find
>> flask (a dependency of the software, which I've put as a
>> propagated-input, and is indeed installed in the container).
>
> Propagated inputs can be inconvenient at times.  I would advise
> looking where requisomatic is referring to flask, and replacing
> flask --> (string-append (assoc-ref inputs "flask") "/bin/flask")
> using substitute*.

OK, I understand this part :) The executable is gunicorn, so I'll just
use its full path in the script that launches the service.

My previous solution was to source the profile beforehand:

https://gitlab.com/edouardklein/requisomatic/-/blob/baf3fe51ad8bbabbcbc467dff92ec02c43e6daf1/guix.scm#L200

I do agree that a rich profile feels dirty and I too don't like
propagated inputs too much, but with Python I don't understand what the
best way to do it is ?

For example, in order to package a Python application in a way that does
not break the host system, I had to put all the search-paths in a
wrapper script and change all propagated inputs as just inputs. I find
the resulting code quite ugly, and I don't think I'm doing things right:
https://gitlab.com/edouardklein/gendscraper/-/blob/980f2597cc36bc79a9be7af5814e0fcf2313b677/gendscraper.scm#L1176

I wouldn't like to have to this for every service. If propagated inputs
are inconvenient, and if services run in an environment where the
search-paths of the installed packages are not available, how can we
easily tell the software where to find its dynamically-loaded parts ?


>
>> But, when I try to run it with shepherd, it fails because it can't find
>> flask (a dependency of the software, which I've put as a
>> propagated-input, and is indeed installed in the container).
>> [...]
>
> Some advice (warning: I'm not familiar with gunicorn or requisomatic
> at all).

I used flask and gunicorn in this project, but I'm not familiar with
them either ;)

>
>> 
>> -----extract from my operating-system declaration file-------
>> (define requisomatic-shepherd-service
>>   ([...] (shepherd-service
>>             [...]
>             (documentation "Run the requisomatic server")
>>             (start #~((make-forkexec-constructor
>>                     ;;   (append
>>                     ;;    (if db-file
>>                     ;;      `("env"
>>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" db-file))
>>                     ;;      '())
>>                                         '("gunicorn" "requisomatic:app")
>
> Normally, services refer by absolute path to the binary to run, and not rely
> on the PATH (the latter would require polluting the system profile).
> Idiomatically, one would write
>
>   '(#$(file-append gunicorn "/bin/gunicorn")
>     #$(file-append requisomatic) "/wherever/the/binary/is")
>
I did not know about file-append even though it is in the manual. Thanks :)


>>                        #:directory (string-append #$requisomatic "/bin/requisomatic/")
>
> Why are you changing the working directory to
> (string-append #$requisomatic "/bin/requisomatic/"),

because . is by default where flask will look for the code it needs.

>and why is "/bin/requisomatic" a
> directory and not an executable?

Because flask/jinja/etc. are normative about where they want stuff to be
and I put everything in a single dir. As most of the assets is code
(some are html templates) I put it in bin/... but I could put them in
/lib of wherever. This is not the cleanest, but I really wanted to get a
guix operating system declaration to work before I did things cleanly.

> Is that a gunicorn thing?

More like flask, but yeah, it's a framework thing.

>
>> Why is the PYTHONPATH (and the other env vars, for that matter) not
>> propagated from the package to the shepherd service by default ?
>
> How is the shepherd service supposed to automagically know which packages
> to include in the environment variables?
>

I realised afterwards that by extending the profile-service-type,
services can say what packages they need (I currently add them to the
operating-system definition). I would expect the shepherd
command to run in an environment where the search-paths of those package
is set. I have no idea if this is complicated to do, but as a service
writer I would enjoy it very much.

>> And how can I make it so ?
>
> Use the #:environment-variables option, see e.g.
> bitlbee-shepherd-service

I see the concept, but the list is hardcoded, and for even moderately
complex Python application (or any other dynamic language) the list of
env variables to set will become huge.

> Or create a wrapper.  See e.g. wrapped-dbus-service.

This looks like a clean version of what I did for gendscraper, but the
problem remains of which variable to set. This information exists in
each individual package. Some will need additions to e.g. PYTHONPATH,
others to LD_LIBRARY_PATH, etc. We need to walk the entire dependency
DAG somehow.
>
>> Follow up question, can shepherd services be specified to run in a
>> specific profile ?
>
> IIUC, currently shepherd services aren't run in *any* profile at all.
> It would be useful to have a function manifest->environment-gexp
> though.

Or package->search-paths ? 

>
>> So that I can have two services with incompatible
>> dependencies running at the same time in the same operating-system ?
> Yes, it with "dependencies" you mean packages, and not other services.
>

I meant packages :)


Thank you very much for your help, it seems you know services inside
out. I can now pinpoint way better what it is that I don't understand. I
hope I was more clear in expressing it.

Cheers !

Edouard.

> Greetings,
> Maxime.