bug#67802: Shepherd is not able to run simple networked programs as services

bug#67802: Shepherd is not able to run simple networked programs as services

[Lars Rustand]

Hello, I have created two very simple shepherd services for two
different mail programs (offlineimap and davmail). Both of the programs
are able to run without problem when ran from the commandline, but both
of them fail with networking related errors when I try to run them as
shepherd services.

I have read all the relevant sections of the manual and looked at
similar shepherd services in the source code, but I can't find anything
that I am missing in my services. I am either missing something obvious
here or shepherd is doing something weird that messes up these programs.

Here is my service definition for offlineimap:

--8<---------------cut here---------------start------------->8---
(define-module (lrustand services offlineimap)
  #:use-module (gnu)
  #:use-module (gnu services)
  #:use-module (gnu packages mail)
  #:use-module (gnu services shepherd)
  #:use-module (gnu services configuration)
  #:use-module (guix gexp)
  #:use-module (guix records)
  #:use-module (ice-9 curried-definitions)
  #:use-module (gnu home services)
  ;; For the 'home-shepherd-service-type' mapping.
  #:use-module (gnu home services shepherd)
  #:export (offlineimap-configuration
            offlineimap-configuration?

            offlineimap-configuration-log-file
            offlineimap-configuration-pid-file

            offlineimap-shepherd-service
            offlineimap-service-type
            home-offlineimap-service-type))

(define-configuration/no-serialization offlineimap-configuration
  (pid-file
   (string "/var/run/offlineimap.pid")
   "Where to store the PID file.")
  (config-file
   (string "/home/lars/.config/offlineimap/config")
   "Configuration file to use.")
  (log-file
   (string "/home/lars/offlineimap.log")
   "File where ‘offlineimap’ writes its log to.")
  (user
   (string "lars")
   "")
  (extra-options
   (list-of-strings '())
   "This option provides an “escape hatch” for the user to provide
arbitrary command-line arguments to ‘offlineimap’ as a list of strings.")
  (home-service?
   (boolean for-home?)
   ""))

(define offlineimap-shepherd-service
  (match-record-lambda <offlineimap-configuration>
    (pid-file config-file log-file user extra-options home-service?)
    (list (shepherd-service
           (provision '(offlineimap))
           (documentation "")
           (requirement (if home-service? '() '(networking user-processes)))
           (start #~(make-forkexec-constructor
                      (list (string-append #$offlineimap
                                           "/bin/offlineimap")
                             #$@extra-options
                             "-c" #$config-file
                             "-l" #$log-file)
                     #:user #$user
                     #:environment-variables
                     (list (string-append "HOME=" (passwd:dir (getpw #$user))))
                     #:log-file #$log-file
                     #:pid-file #$pid-file))
           (stop #~(make-kill-destructor))
           (one-shot? #f)
           (respawn? #t)))))

(define offlineimap-service-type
  (service-type
   (name 'offlineimap)
   (extensions
    (list (service-extension shepherd-root-service-type
                             offlineimap-shepherd-service)))
   (default-value (offlineimap-configuration))
   (description
    "Synchronize remote IMAP mail with local Maildir.")))

(define home-offlineimap-service-type
  (service-type
   (inherit (system->home-service-type offlineimap-service-type))
   (default-value (for-home (offlineimap-configuration)))))
--8<---------------cut here---------------end--------------->8---



And here is my service definition for davmail:


--8<---------------cut here---------------start------------->8---
(define-module (lrustand services davmail)
  #:use-module (gnu)
  #:use-module (gnu services)
  #:use-module (gnu packages mail)
  #:use-module (gnu services shepherd)
  #:use-module (gnu services configuration)
  #:use-module (guix gexp)
  #:use-module (guix records)
  #:use-module (ice-9 curried-definitions)
  #:use-module (gnu home services)
  ;; For the 'home-shepherd-service-type' mapping.
  #:use-module (gnu home services shepherd)
  #:export (davmail-configuration
            davmail-configuration?

            davmail-configuration-log-file
            davmail-configuration-pid-file

            davmail-shepherd-service
            davmail-service-type
            home-davmail-service-type))

(define-configuration/no-serialization davmail-configuration
  (pid-file
   (string "/var/run/davmail.pid")
   "Where to store the PID file.")
  (config-file
   (string "/home/lars/.config/davmail/davmail.properties")
   "Configuration file to use.")
  (log-file
   (string "/home/lars/davmail.log")
   "File where ‘davmail’ writes its log to.")
  (extra-options
   (list-of-strings '())
   "This option provides an “escape hatch” for the user to provide
arbitrary command-line arguments to ‘davmail’ as a list of strings.")
  (home-service?
   (boolean for-home?)
   ""))

(define davmail-shepherd-service
  (match-record-lambda <davmail-configuration>
    (pid-file config-file log-file extra-options home-service?)
    (list (shepherd-service
           (provision '(davmail))
           (documentation "")
           (requirement (if home-service? '() '(user-processes)))
           (start #~(make-forkexec-constructor
                      (list (string-append #$davmail
                                           "/bin/davmail")
                             #$@extra-options
                             "-c" #$config-file)
                     #:log-file #$log-file
                     #:pid-file #$pid-file))
           (stop #~(make-kill-destructor))
           (one-shot? #f)
           (respawn? #t)))))

(define davmail-service-type
  (service-type
   (name 'davmail)
   (extensions
    (list (service-extension shepherd-root-service-type
                             davmail-shepherd-service)))
   (default-value (davmail-configuration))
   (description
    "Gateway for Outlook mail to regular IMAP and SMTP")))

(define home-davmail-service-type
  (service-type
   (inherit (system->home-service-type davmail-service-type))
   (default-value (for-home (davmail-configuration)))))
--8<---------------cut here---------------end--------------->8---


Offlineimap fails with a socket error:

--8<---------------cut here---------------start------------->8---
2023-12-12 19:34:33
2023-12-12 19:34:33 Traceback:
2023-12-12 19:34:33   File "/gnu/store/g2231vkzbnncyj72085n1ylkh8hq16lm-offlineimap-8.0.0/lib/python3.10/site-packages/offlineimap/accounts.py", line 298, in syncrunner
2023-12-12 19:34:33     self.__sync()
2023-12-12 19:34:33   File "/gnu/store/g2231vkzbnncyj72085n1ylkh8hq16lm-offlineimap-8.0.0/lib/python3.10/site-packages/offlineimap/accounts.py", line 374, in __sync
2023-12-12 19:34:33     remoterepos.getfolders()
2023-12-12 19:34:33   File "/gnu/store/g2231vkzbnncyj72085n1ylkh8hq16lm-offlineimap-8.0.0/lib/python3.10/site-packages/offlineimap/repository/IMAP.py", line 681, in getfolders
2023-12-12 19:34:33     imapobj = self.imapserver.acquireconnection()
2023-12-12 19:34:33   File "/gnu/store/g2231vkzbnncyj72085n1ylkh8hq16lm-offlineimap-8.0.0/lib/python3.10/site-packages/offlineimap/imapserver.py", line 683, in acquireconnection
2023-12-12 19:34:33     e.args[0][:35] == 'IMAP4 protocol error: socket error:':
2023-12-12 19:34:33
--8<---------------cut here---------------end--------------->8---


And the davmail service fails with:

--8<---------------cut here---------------start------------->8---
2023-12-12 19:01:01 openjdk version "21" 2023-09-19
2023-12-12 19:01:01 OpenJDK Runtime Environment (build 21+-adhoc.nixbld.source)
2023-12-12 19:01:01 OpenJDK 64-Bit Server VM (build 21+-adhoc.nixbld.source, mixed mode, sharing)
2023-12-12 19:01:01,734 DEBUG [main] davmail.DavGateway  - Start DavMail in server mode
2023-12-12 19:01:01 2023-12-12 19:01:01,734 DEBUG [main] davmail.DavGateway  - Start DavMail in server mode
2023-12-12 19:01:01,759 INFO  [main] davmail  - DavMail Gateway 6.2.0-3464 listening on SMTP port 1025 IMAP port 1143 CALDAV port 1080 LDAP port 1389
2023-12-12 19:01:01 2023-12-12 19:01:01,759 INFO  [main] davmail  - DavMail Gateway 6.2.0-3464 listening on SMTP port 1025 IMAP port 1143 CALDAV port 1080 LDAP port 1389
2023-12-12 19:01:01,828 DEBUG [CheckRelease] davmail.http.HttpClientAdapter  - GET https://davmail.sourceforge.net/version.txt
2023-12-12 19:01:01 2023-12-12 19:01:01,828 DEBUG [CheckRelease] davmail.http.HttpClientAdapter  - GET https://davmail.sourceforge.net/version.txt
2023-12-12 19:01:01,849 DEBUG [CheckRelease] davmail.http.DavGatewaySSLSocketFactory  - createSocket davmail.sourceforge.net 443
2023-12-12 19:01:01 2023-12-12 19:01:01,849 DEBUG [CheckRelease] davmail.http.DavGatewaySSLSocketFactory  - createSocket davmail.sourceforge.net 443
2023-12-12 19:01:02,404 DEBUG [CheckRelease] davmail.DavGateway  - DavMail released version: 6.2.0-3464
2023-12-12 19:01:02 2023-12-12 19:01:02,404 DEBUG [CheckRelease] davmail.DavGateway  - DavMail released version: 6.2.0-3464
2023-12-12 19:01:06,532 DEBUG [Shutdown] davmail  - Stopping DavMail gateway
2023-12-12 19:01:06 2023-12-12 19:01:06,532 DEBUG [Shutdown] davmail  - Stopping DavMail gateway
2023-12-12 19:01:06,539 INFO  [Shutdown] davmail  - DavMail gateway stopped
2023-12-12 19:01:06 2023-12-12 19:01:06,539 INFO  [Shutdown] davmail  - DavMail gateway stopped
--8<---------------cut here---------------end--------------->8---


They both seem to fail when opening an *outgoing* socket, but davmail
seems to be able to start *listening* on several ports just fine.

bug#67802: Shepherd is not able to run simple networked programs as services

[Lars Rustand]

Lars Rustand <rustand.lars@gmail.com> writes:

> Hello, I have created two very simple shepherd services for two
> different mail programs (offlineimap and davmail). Both of the programs
> are able to run without problem when ran from the commandline, but both
> of them fail with networking related errors when I try to run them as
> shepherd services.
...
> They both seem to fail when opening an *outgoing* socket, but davmail
> seems to be able to start *listening* on several ports just fine.

So, I figured this out. It had nothing to do with networking even though
it looked like it. The problem was that I had cargo cult-copied a
#:pid-file parameter from another service, believing that this was just
a path were Shepherd could create a pid-file for the service, but
Shepherd was in fact expecting the program to create the pid-file. So
when the program did not create this pid-file then Shepherd killed the
program.

So the original bug I reported is in fact not a bug at all and can be
closed. However, the error handling in Shepherd could be improved in
order to make it more clear what is happening.

bug#67802: Shepherd is not able to run simple networked programs as services

[Ludovic Courtès]

Hi,

Lars Rustand <rustand.lars@gmail.com> skribis:

> So, I figured this out. It had nothing to do with networking even though
> it looked like it. The problem was that I had cargo cult-copied a
> #:pid-file parameter from another service, believing that this was just
> a path were Shepherd could create a pid-file for the service, but
> Shepherd was in fact expecting the program to create the pid-file. So
> when the program did not create this pid-file then Shepherd killed the
> program.
>
> So the original bug I reported is in fact not a bug at all and can be
> closed. However, the error handling in Shepherd could be improved in
> order to make it more clear what is happening.

Good that you found the issue!  I improved logging for this particular
case as a result of this report:

  https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=ace4a3b2dba43b826640d5c1ca970f9040d27416

Thanks for reporting it,
Ludo’.