where to put helper to send stdout/stderr to syslog?

where to put helper to send stdout/stderr to syslog?

[Robert Vollmert]

Hi all,

I came up with the following wrapper that uses logger to redirect a
program’s console output to syslog:

(define* (logger-wrapper name exec . args)
  "Return a derivation that builds a script to start a process with
standard output and error redirected to syslog via logger."
  (define exp
    #~(begin
        (use-modules (ice-9 popen))
        (let* ((pid  (number->string (getpid)))
               (cmd  (string-append "logger -t " #$name " --id=" pid))
               (log  (open-output-pipe cmd)))
          (dup log 1)
          (dup log 2)
          (execl #$exec #$exec #$@args))))
  (program-file (string-append name "-logger") exp))

I use it as follows for a service that doesn’t support syslog:

(shepherd-service
  ...
  (start #~(make-forkexec-constructor
             '(#$(logger-wrapper “postgrest” (file-append postgrest “/bin/postgrest”)))
             …

Would logger-wrapper be generally useful to have available? If so,
is it named well, and where would it fit?

Cheers
Robert

Re: where to put helper to send stdout/stderr to syslog?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

Hi Robert,

hmm, I like it in principle.

But doesn't shepherd already log to /dev/kmsg and/or /dev/log (so that ends up
in syslog)?  Since exec-command&co keep the standard output and standard error,
they (and thus all shepherd services) should also already log to the
aforementioned syslog by default.

What is the use case you envision?

> Would logger-wrapper be generally useful to have available? If so,
> is it named well, and where would it fit?

I think it could be made part of shepherd and be exported there, then everyone
could use it.  Logging to syslog isn't exactly an obscure requirement :)

Although shepherd already has its own /dev/log (syslog) client implementation,
the external "logger" executable (or similar) is still necessary, because
/dev/log is a UNIX domain socket and one can't write to UNIX domain sockets
the same way one does pipes.  Although it might be possible (and not
advisable) to connect() the socket and then dup it to 1 and 2 for the child :P

P. S. The way you invoke logger (without full path or gexp package reference)
it will pick up a random logger implementation.  I'm surprised that it works
at all that way.

P. S. Your implementation has shell injection because "name" could contain
spaces and/or semicolons.  I suggest not to use the shell command string but
rather passing logger's argv directly.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: where to put helper to send stdout/stderr to syslog?

[Robert Vollmert]

> On 17. Jun 2019, at 14:45, Danny Milosavljevic <dannym@scratchpost.org> wrote:
> 
> But doesn't shepherd already log to /dev/kmsg and/or /dev/log (so that ends up
> in syslog)?  Since exec-command&co keep the standard output and standard error,
> they (and thus all shepherd services) should also already log to the
> aforementioned syslog by default.
> 
> What is the use case you envision?

It sure doesn’t seem like it. For that service, everything lands on
/dev/console, and I even see some sshd messages there that
don’t make it to syslog...

>> Would logger-wrapper be generally useful to have available? If so,
>> is it named well, and where would it fit?
> 
> I think it could be made part of shepherd and be exported there, then everyone
> could use it.  Logging to syslog isn't exactly an obscure requirement :)

Oh I agree that shepherd should in principle deal with capturing
stdout/stderr. I rather like how systemd‘s status command
shows the last few lines of process output. Just not sure this kind
if wrapper fits there.

> P. S. The way you invoke logger (without full path or gexp package reference)
> it will pick up a random logger implementation.  I'm surprised that it works
> at all that way.

I didn’t figure out how else to refer to it. It’s part of bsdutils upstream
as far as I can tell, and installed by default.

> P. S. Your implementation has shell injection because "name" could contain
> spaces and/or semicolons.  I suggest not to use the shell command string but
> rather passing logger's argv directly.

Ok, will try to figure out how to open pipes with explicit argv.

Thanks
Robert

Re: where to put helper to send stdout/stderr to syslog?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2715 bytes --]

Hi Robert,

On Mon, 17 Jun 2019 15:12:27 +0200
Robert Vollmert <rob@vllmrt.net> wrote:

> > On 17. Jun 2019, at 14:45, Danny Milosavljevic <dannym@scratchpost.org> wrote:
> > 
> > But doesn't shepherd already log to /dev/kmsg and/or /dev/log (so that ends up
> > in syslog)?  Since exec-command&co keep the standard output and standard error,
> > they (and thus all shepherd services) should also already log to the
> > aforementioned syslog by default.
> > 
> > What is the use case you envision?  
> 
> It sure doesn’t seem like it. For that service, everything lands on
> /dev/console, and I even see some sshd messages there that
> don’t make it to syslog...

Hmm, could you file a bug report to bug-guix@gnu.org about that?

The intent was that shepherd logs its own stdout & stderr to syslog, and by
process attribute inheritance every service does, too--except when a service
doesn't want to, of course.  I was figuring that maybe the process name that
was logged was wrong in your case or something--but not logging at all is AWFUL.

I've read modules/shepherd/comm.scm now and that's definitely a lot more
complicated than I remember.  I think that duping loses the custom guile
ports we had set up (it's obvious in retrospect).

Using "logger" for everything has the additional wrinkle that when the system
shepherd starts up, syslogd is not started up yet.  The "logger" executable
has NO fallback to /dev/kmsg in order to log into the kernel ring buffer in
the mean time.  (shepherd does have it--but that doesn't help us much in this
case)

If we had something like "logger" but with /dev/kmsg fallback, it would always
work whether or not syslogd was already started up (it would buffer the messages
into the kernel ring buffer /dev/kmsg until syslogd started up, at which point
syslogd would read out the kernel ring buffer and log the value, although with
a strange prefix).

Another possibility is to implement socket activation in shepherd, make it claim
/dev/log and start syslogd once /dev/log is first accessed (on demand). 

Shepherd socket activation would still have to take care of (unintended)
service dependency cycles that could arise, though.

In any case, it will end up using something like your procedure, so I'm OK with
adding a variant of it.
(I don't like the name much--maybe rather "with-syslog-console" or something)

>I didn’t figure out how else to refer to it

(string-append #$inetutils "/bin/logger") or something like that.

As an aside, interesting that inetutils' logger doesn't use glibc's openlog.
inetutils wrote their own implementation.

glibc's openlog doesn't fall back to /dev/kmsg either.  Grr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: where to put helper to send stdout/stderr to syslog?

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I think it could be made part of shepherd and be exported there, then everyone
> could use it.  Logging to syslog isn't exactly an obscure requirement :)

+1!

> Although shepherd already has its own /dev/log (syslog) client implementation,
> the external "logger" executable (or similar) is still necessary, because
> /dev/log is a UNIX domain socket and one can't write to UNIX domain sockets
> the same way one does pipes.  Although it might be possible (and not
> advisable) to connect() the socket and then dup it to 1 and 2 for the child :P

Yes, that should be enough.

Robert, would you like to give it a go?

Thanks,
Ludo’.

Re: where to put helper to send stdout/stderr to syslog?

[Robert Vollmert]

> On 18. Jun 2019, at 15:32, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> Hi,
> 
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
> 
>> I think it could be made part of shepherd and be exported there, then everyone
>> could use it.  Logging to syslog isn't exactly an obscure requirement :)
> 
> +1!
> 
>> Although shepherd already has its own /dev/log (syslog) client implementation,
>> the external "logger" executable (or similar) is still necessary, because
>> /dev/log is a UNIX domain socket and one can't write to UNIX domain sockets
>> the same way one does pipes.  Although it might be possible (and not
>> advisable) to connect() the socket and then dup it to 1 and 2 for the child :P
> 
> Yes, that should be enough.
> 
> Robert, would you like to give it a go?

I’d rather not get too deep into shepherd at this point.

Here’s the wrapper updated in response to Danny’s comments, in case you want to
reuse that inside shepherd:

(define* (logger-wrapper name exec . args)
  "Return a derivation that builds a script to start a process with
standard output and error redirected to syslog via logger."
  (define exp
    #~(begin
        (use-modules (ice-9 popen))
        (let* ((pid    (number->string (getpid)))
               (logger #$(file-append inetutils "/bin/logger"))
               (args   (list "-t" #$name (string-append "--id=" pid)))
               (pipe   (apply open-pipe* OPEN_WRITE logger args)))
          (dup pipe 1)
          (dup pipe 2)
          (execl #$exec #$exec #$@args))))
  (program-file (string-append name "-logger") exp))

Cheers
Robert