Shepherd log rotation service

Shepherd log rotation service

[Ludovic Courtès]

Hello Guix!

I’ve just pushed a simple log rotation service for Shepherd:

  https://git.savannah.gnu.org/cgit/shepherd.git/commit/?h=devel&id=0484726801c2b5c1b7deecb9a96054c2c510ac71

It rotates log files roughly the same way we’ve been doing since the
70’s, but there’s a couple of advantages compared to what we’re
currently doing in Guix System:

  • No need to repeat the name of log files since shepherd already knows
    them via #:log-file (that’s another reason to avoid
    non-shepherd-managed log files).

  • Rotation is race-free: it’s impossible to lose a line of log while
    the file is being rotated.  This is guaranteed by the “logger”,
    which offers a method to atomically close its log file, rename it,
    and open a new empty log file.

  • The log rotation service is a timer so one can inspect it with ‘herd
    status log-rotation’, trigger it with ‘herd trigger log-rotation’,
    and so on.

At this point the log file of shepherd itself, for instance
/var/log/messages, is not handled; this will have to be fixed.

There aren’t many options to specify how to rotate logs (very few
compared to rottlog!), but I figured we’d rather have something simple
that works well and without surprises; we can always add knobs later if
necessary.

Feedback welcome!

Ludo’.

Re: Shepherd log rotation service

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Ludo',

> I’ve just pushed a simple log rotation service for Shepherd:

Wow, that's another cool addition!  I am currently testing a patch that
removes the rottlog service extension from all services that use
#:log-file.  Not all do.

I should will make an immediate dent in my 200 MB log file for Fcgiwrap,
but not for an Rspamd file of similar size.  Also, my Nginx log file
with 1.3 GB will remain unaffected.  Most significantly, it simplifies a
lot of code.

To make the addition of my new log-rotation-service-type compatible with
folks who use the rottlog service extension outside the main repo, I
also hope to make the service extension a no-op when the underlying
Shepherd service uses the #:log-file facility.

Meanwhile, I have a few questions:

Should log-rotation-service be called make-log-rotation-service?

Should the key #:rotation-size-threshold be renamed to size-threshold?
I'm not asking to change the name of the variable holding the default
value.  The current key looks longer than needed in this code:

(define (log-rotation-shepherd-service config)
  (match-record
      config <log-rotation-configuration> (compression
                                           event
                                           expiry
                                           size-threshold)
    #~((log-rotation-service #$event
                             #:compression #$compression
                             #:expiry #$expiry
                             #:rotation-size-threshold #$size-threshold))))

Also, is it good practice, in log-rotation-service, to use an optional
argument (for event) together with keyword arguments?  Why not use
keyword arguments all around?

Finally, are there truly no (Shepherd) service requirements?  Or will
any Shepherd service that uses #:log-file automatically make sure that
the file-system is available?

Thank you for this cool new feature!

Kind regards
Felix




On Sat, May 18 2024, Ludovic Courtès wrote:

> Hello Guix!
>
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?h=devel&id=0484726801c2b5c1b7deecb9a96054c2c510ac71
>
> It rotates log files roughly the same way we’ve been doing since the
> 70’s, but there’s a couple of advantages compared to what we’re
> currently doing in Guix System:
>
>   • No need to repeat the name of log files since shepherd already knows
>     them via #:log-file (that’s another reason to avoid
>     non-shepherd-managed log files).
>
>   • Rotation is race-free: it’s impossible to lose a line of log while
>     the file is being rotated.  This is guaranteed by the “logger”,
>     which offers a method to atomically close its log file, rename it,
>     and open a new empty log file.
>
>   • The log rotation service is a timer so one can inspect it with ‘herd
>     status log-rotation’, trigger it with ‘herd trigger log-rotation’,
>     and so on.
>
> At this point the log file of shepherd itself, for instance
> /var/log/messages, is not handled; this will have to be fixed.
>
> There aren’t many options to specify how to rotate logs (very few
> compared to rottlog!), but I figured we’d rather have something simple
> that works well and without surprises; we can always add knobs later if
> necessary.
>
> Feedback welcome!
>
> Ludo’.

Re: Shepherd log rotation service

[Ludovic Courtès]

Hi,

Felix Lechner <felix.lechner@lease-up.com> skribis:

> I should will make an immediate dent in my 200 MB log file for Fcgiwrap,
> but not for an Rspamd file of similar size.  Also, my Nginx log file
> with 1.3 GB will remain unaffected.  Most significantly, it simplifies a
> lot of code.

Right.  I think we should augment ‘log-rotation-service’ so it can also
take care of “detached logs” (logs not created by #:log-file).  This is
necessary for nginx and others.

> Should log-rotation-service be called make-log-rotation-service?

I don’t think so.  :-)  This is consistent with ‘repl-service’ and
‘monitoring-service’.

> Should the key #:rotation-size-threshold be renamed to size-threshold?
> I'm not asking to change the name of the variable holding the default
> value.  The current key looks longer than needed in this code:

Hmm, not sure about this one.

> Also, is it good practice, in log-rotation-service, to use an optional
> argument (for event) together with keyword arguments?  Why not use
> keyword arguments all around?

It’s good practice—there’s no ambiguity at the language level.

Whether it’s good style is a different story.  My thinking here is that
‘event’ is the “most important” parameter, hence this special treatment.

> Finally, are there truly no (Shepherd) service requirements?  Or will
> any Shepherd service that uses #:log-file automatically make sure that
> the file-system is available?

Currently we use #:log-file in Guix System without worrying specifying
any ‘file-system-’ requirement.  The assumption is that /var/log is on
the root file system.

Dropping this assumption would complicate things…

(This is unrelated to the log rotation service though.)

Thanks,
Ludo’.

Re: Shepherd log rotation service

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Ludo'

On Sat, May 18 2024, Ludovic Courtès wrote:

> I’ve just pushed a simple log rotation service for Shepherd:

Sorry to double up on my messages.  I might further rename the somewhat
generic sounding %default-calendar-event to %default-rotation-event, or
perhaps even %default-rotation-calendar-event.

Also, the #:compression parameter does not appear to be passed on to
rotate-service-logs here [1] meaning it'll always use 'gzip.

Kind regards
Felix

[1] https://git.savannah.gnu.org/cgit/shepherd.git/tree/modules/shepherd/service/log-rotation.scm?h=devel&id=0484726801c2b5c1b7deecb9a96054c2c510ac71#n204

Re: Shepherd log rotation service

[Ludovic Courtès]

Hi Felix,

Felix Lechner <felix.lechner@lease-up.com> skribis:

> Sorry to double up on my messages.  I might further rename the somewhat
> generic sounding %default-calendar-event to %default-rotation-event, or
> perhaps even %default-rotation-calendar-event.

Good idea, I’ll do that.

> Also, the #:compression parameter does not appear to be passed on to
> rotate-service-logs here [1] meaning it'll always use 'gzip.

Oops, will push a fix for that too.

Thanks for the enthusiasm and for the bug reports!

Ludo’.

Re: Shepherd log rotation service

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Ludo',

On Sat, May 18 2024, Ludovic Courtès wrote:

> I’ve just pushed a simple log rotation service for Shepherd:

I have a patch removing rottlog from all Guix "services" that use the
Shepherd's :log-file feature, which works fine.  Do I need to provide a
Guix "service" for the log rotation?

If so, may I somehow reuse the Shepherd service definition [1] which
doesn't contain any G-exps, in the Guix "service" definition, which
does?

This won't work:

--8<---------------cut here---------------start------------->8---
    (define (log-rotation-shepherd-service config)
      (match-record
          config <log-rotation-configuration> (compression
                                               calendar-event
                                               expiry
                                               size-threshold)
    
        #~((log-rotation-service #$calendar-event
            #:compression #$compression
            #:expiry #$expiry
            #:rotation-size-threshold #$size-threshold))))
--8<---------------cut here---------------end--------------->8---
    
Thank you for your help, and thanks for the recent syslog announcement!

Kind regards
Felix

[1] https://git.savannah.gnu.org/cgit/shepherd.git/tree/modules/shepherd/service/log-rotation.scm?h=devel#n219

Re: Shepherd log rotation service

[Ludovic Courtès]

Hi Felix,

Felix Lechner <felix.lechner@lease-up.com> skribis:

> On Sat, May 18 2024, Ludovic Courtès wrote:
>
>> I’ve just pushed a simple log rotation service for Shepherd:
>
> I have a patch removing rottlog from all Guix "services" that use the
> Shepherd's :log-file feature, which works fine.  Do I need to provide a
> Guix "service" for the log rotation?

Nice, you’re ahead of me! :-)

Here’s how you can provide the log rotation service in your Guix System
config:

--8<---------------cut here---------------start------------->8---
(define log-rotation-service
  (simple-service 'shepherd-log-rotation
                  shepherd-root-service-type
                  (list (shepherd-service
                         (provision '(log-rotation))
                         (modules '((shepherd service log-rotation)))
                         (free-form #~(log-rotation-service))))))


(operating-system
  ;; …
  (services (append (list log-rotation-service) …)))
--8<---------------cut here---------------end--------------->8---

From there you can check ‘herd files log-rotation’ etc.

(In my config I was too lazy to remove rottlog, so I actually have
both…)

Ludo’.