cron-service

cron-service

[Danny Milosavljevic]

Hi,

I've seen the mcron package but I don't understand how it gets launched. Should there be a cron-service ?

How about the configuration which cron jobs to run? mcron can do both the traditional format and S-Expression format, so it would be possible to directly have the system cronjobs inside /etc/config.scm as an S-Expression. Does that work?

Re: cron-service

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I've seen the mcron package but I don't understand how it gets launched. Should there be a cron-service ?

Ah ha!  I have a preliminary mcron service (attached).  It’s
undocumented and subject to change, but feedback is welcome!

Currently it’s designed to run on ‘mcron’ process per user/group pair.
Eventually, we’ll probably change mcron to allow us to run a single
instance as root, and it will automatically setuid/setgid for each job.

I told Mathieu Lirzin off-line about a couple of minor issues that would
need to be fixed in his mcron branch¹, after which we can probably
commit it (but let’s not put pressure on him!).

Thanks,
Ludo’.

¹ https://notabug.org/mthl/mcron


[-- Attachment #2: mcron service --]
[-- Type: text/plain, Size: 5370 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (gnu services mcron)
  #:use-module (gnu services)
  #:use-module (gnu services base)
  #:use-module (gnu services shepherd)
  #:autoload   (gnu packages guile) (mcron)
  #:use-module (guix records)
  #:use-module (guix gexp)
  #:use-module (srfi srfi-1)
  #:use-module (ice-9 match)
  #:use-module (ice-9 vlist)
  #:export (mcron-configuration
            mcron-configuration?
            mcron-configuration-mcron
            mcron-configuration-jobs

            mcron-job
            mcron-job?
            mcron-job-user
            mcron-job-group
            mcron-job-specification
            mcron-job-imported-modules
            mcron-job-modules

            mcron-service-type
            mcron-service))

;;; Commentary:
;;;
;;; This module implements a service that to run instances of GNU mcron, a
;;; periodic job execution daemon.  Example of a service:
;;
;; (service mcron-service-type
;;          (mcron-configuration
;;           (jobs (list (mcron-job
;;                        (user "alice")
;;                        (specification
;;                         #~(job next-second-from
;;                                (lambda ()
;;                                  (call-with-output-file "/dev/console"
;;                                    (lambda (port)
;;                                      (display "hello!\n" port)))))))))))
;;;
;;; Code:

(define-record-type* <mcron-configuration> mcron-configuration
  make-mcron-configuration
  mcron-configuration?
  (mcron             mcron-configuration-mcron    ;package
                     (default mcron))
  (jobs              mcron-configuration-jobs     ;list of <mcron-job>
                     (default '())))

(define-record-type* <mcron-job> mcron-job make-mcron-job
  mcron-job?
  (user              mcron-job-user (default "root"))  ;string
  (group             mcron-job-group (default #f))     ;string | #f
  (specification     mcron-job-specification)          ;gexp
  (imported-modules  mcron-job-imported-modules        ;list
                     (default '()))
  (modules           mcron-job-modules                 ;list
                     (default '())))

(define (job-file job)
  (scheme-file "mcron-job"
               (mcron-job-specification job)))

(define (mcron-shepherd-service mcron jobs)
  (match jobs
    ((($ <mcron-job> user group) _ ...)
     (shepherd-service
      (provision (list (string->symbol
                        (string-append "mcron-" user
                                       (if group
                                           (string-append "-" group)
                                           "")))))
      (requirement '(user-processes))
      (start #~(make-forkexec-constructor
                (list (string-append #$mcron "/bin/mcron")
                      #$@(map job-file jobs))
                #:user #$user
                #:group #$(if user
                              (or group
                                  #~(group:name
                                     (getgrgid (passwd:gid (getpw #$user)))))
                              group)))
      (stop #~(make-kill-destructor))))))

(define mcron-shepherd-services
  (match-lambda
    (($ <mcron-configuration> mcron jobs)
     (define sorted-jobs
       (fold (lambda (job result)
               (match job
                 (($ <mcron-job> user group)
                  (vhash-cons (list user group) job result))))
             vlist-null
             jobs))

     (define users+groups
       (delete-duplicates
        (match jobs
          ((($ <mcron-job> users groups) ...)
           (zip users groups)))))

     (map (lambda (key)
            (mcron-shepherd-service mcron (vhash-fold* cons '() key sorted-jobs)))
          users+groups))))

(define mcron-service-type
  (service-type (name 'mcron)
                (extensions
                 (list (service-extension shepherd-root-service-type
                                          mcron-shepherd-services)))
                (compose concatenate)
                (extend (lambda (config jobs)
                          (mcron-configuration
                           (inherit config)
                           (jobs (append (mcron-configuration-jobs config)
                                         jobs)))))))

(define* (mcron-service #:optional (mcron mcron))
  (service mcron-service-type (mcron-configuration (mcron mcron))))

;;; mcron.scm ends here

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Type: text/x-patch, Size: 3930 bytes --]

modified   gnu/packages/guile.scm
@@ -41,6 +41,7 @@
   #:use-module (gnu packages ed)
   #:use-module (gnu packages base)
   #:use-module (gnu packages texinfo)
+  #:use-module (gnu packages man)
   #:use-module (gnu packages gettext)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages python)
@@ -424,6 +425,54 @@ Guile, so its configuration can be written in Scheme; the original cron
 format is also supported.")
     (license gpl3+)))
 
+(define-public mcron2
+  (let ((commit "573a09a32684c091cb8e8f521946f8bf90a295af"))
+    (package
+      (inherit mcron)
+      (name "mcron2")
+      (version (string-append (package-version mcron) "-0."
+                              (string-take commit 7)))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://notabug.org/mthl/mcron/")
+                      (commit commit)))
+                (sha256
+                 (base32
+                  "0m4kcpxmcr3rf6l6dd2z0m427gc2y1xx9z361j2zw3jgvamg0yhw"))
+                (file-name (string-append name "-" version "-checkout"))))
+      (native-inputs
+       `(("autoconf" ,autoconf)
+         ("automake" ,automake)
+         ("pkg-config" ,pkg-config)
+         ("texinfo" ,texinfo)
+         ("help2man" ,help2man)))
+      (arguments
+       `(#:modules ((ice-9 match) (ice-9 ftw)
+                    ,@%gnu-build-system-modules)
+
+         #:phases (modify-phases %standard-phases
+                    (add-after 'unpack 'bootstrap
+                      (lambda _
+                        (zero? (system* "autoreconf" "-vfi"))))
+                    (add-after 'install 'wrap-mcron
+                      (lambda* (#:key outputs #:allow-other-keys)
+                        ;; Wrap the 'mcron' command to refer to the right
+                        ;; modules.
+                        (let* ((out  (assoc-ref outputs "out"))
+                               (bin  (string-append out "/bin"))
+                               (site (string-append
+                                      out "/share/guile/site")))
+                          (match (scandir site)
+                            (("." ".." version)
+                             (let ((modules (string-append site "/" version)))
+                               (wrap-program (string-append bin "/mcron")
+                                 `("GUILE_LOAD_PATH" ":" prefix
+                                   (,modules))
+                                 `("GUILE_LOAD_COMPILED_PATH" ":" prefix
+                                   (,modules)))
+                               #t))))))))))))
+
 (define-public guile-lib
   (package
     (name "guile-lib")
modified   gnu/system/examples/bare-bones.tmpl
@@ -2,7 +2,7 @@
 ;; for a "bare bones" setup, with no X11 display server.
 
 (use-modules (gnu))
-(use-service-modules networking ssh)
+(use-service-modules networking ssh mcron)
 (use-package-modules admin)
 
 (operating-system
@@ -42,6 +42,15 @@
 
   ;; Add services to the baseline: a DHCP client and
   ;; an SSH server.
-  (services (cons* (dhcp-client-service)
-                   (lsh-service #:port-number 2222)
+  (services (cons* (service mcron-service-type
+                            (mcron-configuration
+                             (jobs (list (mcron-job
+                                          (user "alice")
+                                          (specification
+                                           #~(job next-second-from
+                                                  (lambda ()
+                                                    (call-with-output-file
+                                                        "/dev/console"
+                                                      (lambda (port)
+                                                        (display "hello!\n" port)))))))))))
                    %base-services)))

fstrim and SSDs and cron; was: Re: cron-service

[Danny Milosavljevic]

Hi Ludo,

thanks. 

I can confirm that this works!

I'm using it to regularily call fstrim (one has to regularily call fstrim on SSDs) so the mcron behaviour of, if an appointment was missed, doing it at the first next availability is good for regularity. The man page says that trimming once a week is good.

Longer term maybe there should be a way of specifying that a filesystem should be trimmed in a "file-system" specification. Note that fstrim expects the filesystem to be mounted for it to work.

Re: fstrim and SSDs and cron; was: Re: cron-service

[Ludovic Courtès]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I can confirm that this works!

Great, thanks for testing.

> I'm using it to regularily call fstrim (one has to regularily call
> fstrim on SSDs) so the mcron behaviour of, if an appointment was
> missed, doing it at the first next availability is good for
> regularity. The man page says that trimming once a week is good.
>
> Longer term maybe there should be a way of specifying that a
> filesystem should be trimmed in a "file-system" specification. Note
> that fstrim expects the filesystem to be mounted for it to work.

I had never thought of it, but maybe it’s a good idea.  How do other
distros handle it?

Thanks,
Ludo’.

Re: fstrim and SSDs and cron; was: Re: cron-service

[Thompson, David]

On Tue, May 17, 2016 at 5:24 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
>> I can confirm that this works!
>
> Great, thanks for testing.
>
>> I'm using it to regularily call fstrim (one has to regularily call
>> fstrim on SSDs) so the mcron behaviour of, if an appointment was
>> missed, doing it at the first next availability is good for
>> regularity. The man page says that trimming once a week is good.
>>
>> Longer term maybe there should be a way of specifying that a
>> filesystem should be trimmed in a "file-system" specification. Note
>> that fstrim expects the filesystem to be mounted for it to work.
>
> I had never thought of it, but maybe it’s a good idea.  How do other
> distros handle it?

FWIW, I use SSDs in all my computers and have never run fstrim.  Feels
a lot like the "defragment your hard drive" days.

- Dave

Re: fstrim and SSDs and cron; was: Re: cron-service

[Danny Milosavljevic]

Hi,

> FWIW, I use SSDs in all my computers and have never run fstrim.  Feels
> a lot like the "defragment your hard drive" days.

If you would run it now then it would free many gigabytes.

What it does and why it's needed:

Traditionally filesystems on hard disks would just reuse sectors once they are not needed by the filesystem anymore. For example to delete a file it would just remove the filename and update the usage bitmap metadata. That's it. It wouldn't do anything to the payload sectors. Later on, the payload sector could maybe be used for another file's payload.

For SSDs this is bad since they are trying to do wear levelling. SSDs work by providing an "hard-drive-like" interface to (many) flash blocks in hardware. A flash block can be written (actually: erased) only a limited number of times (< 5000). So there's an algorithm in the drive fireware which makes sure to evenly use all the flash blocks. If you keep writing the same sector, it will eventually move and remap it to another formerly-empty flash block.

The problem with that is the "empty" in the last sentence. Which one is empty? As I said, the traditional filesystem wouldn't do anything to mark the payload setors as empty on the drive itself.

Therefore, a command called "TRIM" was added to the SATA protocol (and other things). fstrim will send the command to the drive, informing it about sectors that are actually unused now. In this way the SSD has more possible targets to move stuff to.

if you don't do it at all, the SSD will eventually run out of spare blocks and won't move stuff away anymore - which means one of the blocks is going to fail very soon (the ones that are written often - which are probably filesystem metadata blocks - maybe the usage bitmap). 

Of course it can still shuffle around used blocks - and probably does. Otherwise it would turn out to be very bad.

There's also the "discard" mount option which automatically trims - but documentation says that it's caused longevity problems in the past for some drives (probably because it sends the TRIM command too often).

Re: fstrim and SSDs and cron; was: Re: cron-service

[Danny Milosavljevic]

> I had never thought of it, but maybe it’s a good idea.  How do other
> distros handle it?

On <https://askubuntu.com/questions/18903/how-to-enable-trim> they say:
>Ubuntu 14.10 onwards:
>In Ubuntu 14.10 and 15.04, TRIMming happens automatically every week on all SSDs supported by fstrim.
>$ tail -n1 /etc/cron.weekly/fstrim 
>/sbin/fstrim --all || true




>Automatic TRIM (Deprecated, Slow):
> [fstab...]

>Encrypted Filesystems
>The last step is not enough though. As long as LUKS is not aware that you want to use TRIM it will effectively block all TRIM operations coming from the LVM partition's file system, for security reasons. Add discard parameter to the cryptdevice options in /etc/crypttab to make LUKS accept the discard behavior of the LVM partition.
>sda5_crypt UUID=e364d03f-[...]6cd7e none luks,discard

More info about the latter:

http://blog.neutrino.es/2013/howto-properly-activate-trim-for-your-ssd-on-linux-fstrim-lvm-and-dmcrypt/

Note that guix /gnu/store/8vg9124cgm8d36zsy9ldcw53b1vrfy2w-lvm2-2.02.109/etc/lvm/lvm.conf explicitly disables (!) "discard".

Re: fstrim and SSDs and cron; was: Re: cron-service

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2472 bytes --]

On Wed, May 18, 2016 at 07:21:18AM +0200, Danny Milosavljevic wrote:
> > I had never thought of it, but maybe it’s a good idea.  How do other
> > distros handle it?
> 
> On <https://askubuntu.com/questions/18903/how-to-enable-trim> they say:
> >Ubuntu 14.10 onwards:
> >In Ubuntu 14.10 and 15.04, TRIMming happens automatically every week on all SSDs supported by fstrim.
> >$ tail -n1 /etc/cron.weekly/fstrim 
> >/sbin/fstrim --all || true
> 
> 
> 
> 
> >Automatic TRIM (Deprecated, Slow):
> > [fstab...]
> 
> >Encrypted Filesystems
> >The last step is not enough though. As long as LUKS is not aware that you want to use TRIM it will effectively block all TRIM operations coming from the LVM partition's file system, for security reasons. Add discard parameter to the cryptdevice options in /etc/crypttab to make LUKS accept the discard behavior of the LVM partition.
> >sda5_crypt UUID=e364d03f-[...]6cd7e none luks,discard
> 
> More info about the latter:
> 
> http://blog.neutrino.es/2013/howto-properly-activate-trim-for-your-ssd-on-linux-fstrim-lvm-and-dmcrypt/
> 
> Note that guix /gnu/store/8vg9124cgm8d36zsy9ldcw53b1vrfy2w-lvm2-2.02.109/etc/lvm/lvm.conf explicitly disables (!) "discard".
> 

I have an SSD in my netbook, and I've had it a bit more than 4 years
now. According to the smart data, it's been powered on for ~18,000
hours, and recently I saw the drive life percentage used counter tick
from 2 to 3%, somewhere around 90 full disk writes.

When changing data:
Read block A, Write block B.

With "dirty" block B:
Read block A, Scrub block B, Write block B.

Trim goes and scrubs all the now unused blocks so that they're ready for
use. In many cases (aka not my machine, which is cpu bound on
reads/writes) reading/writing to an SSD is limited by the SATA port's
speed and the HDD based protocol. By trimming regularly (for me
somewhere between weekly and monthly) you can save the bandwidth-limited
transactions to the SSD for actual writes if you trim ahead of time.

If we wanted to automatically trim for people without making mcron and
its service part of %base-packages or %desktop-services we could add it
as a hook before/after a `guix pull', but that seems too intrusive to
me.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

mcron service

[Ludovic Courtès]

Hi!

Commit c311089b0b19f094e44d3f858c29f77d757332d1 adds ‘mcron-service’.
If you want to give it a try, your feedback is welcome!  Excerpt from
the manual below.

Ludo’.

7.2.7.2 Scheduled Job Execution
...............................

The ‘(gnu services mcron)’ module provides an interface to GNU mcron, a
daemon to run jobs at scheduled times (*note (mcron)Top::).  GNU mcron
is similar to the traditional Unix ‘cron’ daemon; the main difference is
that it is implemented in Guile Scheme, which provides a lot of
flexibility when specifying the scheduling of jobs and their actions.

   For example, to define an operating system that runs the ‘updatedb’
(*note (find)Invoking updatedb::) and the ‘guix gc’ commands (*note
Invoking guix gc::) daily:

     (use-modules (guix) (gnu) (gnu services mcron))

     (define updatedb-job
       ;; Run 'updatedb' at 3 AM every day.
       #~(job '(next-hour '(3))
              "updatedb --prunepaths='/tmp /var/tmp /gnu/store'"))

     (define garbage-collector-job
       ;; Collect garbage 5 minutes after midnight every day.
       #~(job "5 0 * * *"            ;Vixie cron syntax
              "guix gc -F 1G"))

     (operating-system
       ;; ...
       (services (cons (mcron-service (list garbage-collector-job
                                            updatedb-job))
                       %base-services)))

   *Note mcron job specifications: (mcron)Guile Syntax, for more
information on mcron job specifications.  Below is the reference of the
mcron service.

 -- Scheme Procedure: mcron-service JOBS [#:mcron MCRON2]
     Return an mcron service running MCRON that schedules JOBS, a list
     of gexps denoting mcron job specifications.

     This is a shorthand for:
            (service mcron-service-type
                     (mcron-configuration (mcron mcron) (jobs jobs)))

 -- Scheme Variable: mcron-service-type
     This is the type of the ‘mcron’ service, whose value is an
     ‘mcron-configuration’ object.

     This service type can be the target of a service extension that
     provides it additional job specifications (*note Service
     Composition::).  In other words, it is possible to define services
     that provide addition mcron jobs to run.

 -- Data Type: mcron-configuration
     Data type representing the configuration of mcron.

     ‘mcron’ (default: MCRON2)
          The mcron package to use.

     ‘jobs’
          This is a list of gexps (*note G-Expressions::), where each
          gexp corresponds to an mcron job specification (*note mcron
          job specifications: (mcron)Syntax.).

Re: mcron service

[Danny Milosavljevic]

Hi,

I tried it out and can confirm that it still works. Nice!

Just curious, how would I specify the user to run the job under now?

How do I read the guix texinfo manual (using info) when the manual is not installed (i.e. read directly from the git checkout)?

(Also, I wonder whether it would make any sense to be able to specify a list of packages that have to be installed because of the scheduled job(s). Otherwise it could for example be that updatedb isn't available, right?)

Re: mcron service

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I tried it out and can confirm that it still works. Nice!

Thanks for testing!

> Just curious, how would I specify the user to run the job under now?

Using Mathieu’s mcron branch (the ‘mcron2’ package, which is the default
here), this is done via #:user, so:

  #~(job '(next-hour '(1))
          do-something
          #:user "danny")

I’ll add an example of this in the manual.

> How do I read the guix texinfo manual (using info) when the manual is not installed (i.e. read directly from the git checkout)?

Using the standalone Info reader, you can run:

  info -f ./doc/guix.info

and using Emacs:

  C-u C-h i ./doc/guix.info RET

> (Also, I wonder whether it would make any sense to be able to specify a list of packages that have to be installed because of the scheduled job(s). Otherwise it could for example be that updatedb isn't available, right?)

I kept this example simple, but if you want to make sure it finds
‘updatedb’, you can write it as:

  #~(job …
         (string-append #$findutils "/bin/updatedb --foo"))

And if you want to avoid issues related to shell quoting:

  #~(job …
         (lambda ()
           (execl (string-append #$findutils "/bin/updatedb")
                  "updatedb" "--foo")))

HTH!

Ludo’.

Re: mcron service

[Alex Kost]

Ludovic Courtès (2016-06-24 15:00 +0300) wrote:

> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
>> How do I read the guix texinfo manual (using info) when the manual is not installed (i.e. read directly from the git checkout)?
>
> Using the standalone Info reader, you can run:
>
>   info -f ./doc/guix.info
>
> and using Emacs:
>
>   C-u C-h i ./doc/guix.info RET

Or you can just press "I" on the .info file in a dired buffer.

This "I" key (bound to 'dired-info') becomes available when you use
'dired-x'.  I personally have the following in my ".emacs":

  (with-eval-after-load 'dired
    (require 'dired-x))

although the manual recommends another way:
(info "(dired-x) Installation")

-- 
Alex