[bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service

[bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service

[Richard Sent]

Hi all!

This is the previously promised [1] patch series relating to
restic-backup-service. The highlights:

1. init? was added that bootstraps a new backup.

2. RESTIC_PASSWORD_COMMAND support.

3. Add extra-packages field for jobs.

4. Move restic from restic-backup-job to restic-backup-configuration.

5. Add a basic system-level test suite.

Regarding 4, the previous way felt a bit orthogonal to how services of a
similar nature work. (For example, you don't specify an mcron package for
every job.) I can't think of a use case for mixing-and-matching restic
packages with different jobs. If there is a reason, I think a good compromise
would be a "restic-override" job field.

I'm marking this as WIP because there are a couple more things I want to
investigate, but it feels complete enough to submit for feedback. My main
point of interest is what can be done to improve support for Restic's other
backends, see [2]. I imagine 0.9.6 only supports a subset of those, but still.

In my dream world we could combine this patch with an upgrade to the restic
package itself (which, among other things, supports compression). I asked the
developers about (re)including a vendor directory in their tarball but haven't
heard back [3]. Perhaps this service can spark some action on an old Guix
tracking issue [4].

[1]: https://issues.guix.gnu.org/69513#7
[2]: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
[3]: https://github.com/restic/restic/issues/3945#issuecomment-2141090355
[4]: https://issues.guix.gnu.org/63019

Richard Sent (5):
  services: backup: Support bootstrapping an initial restic backup
  services: backup: Add password-command support to restic-service
  services: backup: Add extra-packages field to restic-backup-job
  services: backup: Move restic package to restic-configuration
  tests: Add restic system test.

 doc/guix.texi           |  21 ++++++-
 gnu/services/backup.scm | 131 +++++++++++++++++++++++++++++-----------
 gnu/tests/restic.scm    | 124 +++++++++++++++++++++++++++++++++++++
 3 files changed, 238 insertions(+), 38 deletions(-)
 create mode 100644 gnu/tests/restic.scm


base-commit: a575d0f5d5322bac977423b6bd2742c8dc5a14a6
-- 
2.45.1

[bug#71639] [PATCH WIP 1/5] services: backup: Support bootstrapping an initial restic backup

[Richard Sent]

* gnu/services/backup.scm: (restic-backup-job): Add init? field.
(restic-backup-job-program): Initialize repository if init? is set and
repository does not already exist.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d
---
 doc/guix.texi           |  4 ++++
 gnu/services/backup.scm | 20 ++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0102fd0fad..63c9cbd1a7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41353,6 +41353,10 @@ Miscellaneous Services
 The list of files or directories to be backed up.  It must be a list of
 values that can be lowered to strings.
 
+@item @code{init?} (default: @code{#f}) (type: boolean)
+Whether restic-backup-service should check and (if it does not exist)
+initialize the repository before running the backup.
+
 @item @code{verbose?} (default: @code{#f}) (type: boolean)
 Whether to enable verbose output for the current backup job.
 
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 555e9fc959..eeef11eae7 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2024 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -38,6 +39,7 @@ (define-module (gnu services backup)
             restic-backup-job-password-file
             restic-backup-job-schedule
             restic-backup-job-files
+            restic-backup-job-init?
             restic-backup-job-verbose?
             restic-backup-job-extra-flags
 
@@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job
   (verbose?
    (boolean #f)
    "Whether to enable verbose output for the current backup job.")
+  (init?
+   (boolean #f)
+   "Whether to attempt to initialize a new repository for automated bootstrap purposes.")
   (extra-flags
    (list-of-lowerables '())
    "A list of values that are lowered to strings.  These will be passed as
@@ -118,6 +123,8 @@ (define (restic-backup-job-program config)
          (restic-backup-job-files config))
         (extra-flags
          (restic-backup-job-extra-flags config))
+        (init?
+         (restic-backup-job-init? config))
         (verbose
          (if (restic-backup-job-verbose? config)
              '("--verbose")
@@ -130,6 +137,19 @@ (define (restic-backup-job-program config)
          (setenv "RESTIC_PASSWORD"
                  (with-input-from-file #$password-file read-line))
 
+         (when #$init?
+           ;; Check if the repository exists. See
+           ;; https://github.com/restic/restic/issues/1690 and
+           ;; https://github.com/NixOS/nixpkgs/pull/307962.
+           ;;
+           ;; XXX: restic returns values other than 1 on failure. Use
+           ;; unless EXIT_SUCCESS instead of when EXIT_FAILURE.
+           (unless (equal? EXIT_SUCCESS (system* #$restic "cat" "config"
+                                               "-r" #$repository))
+             ;; Initialize it.
+             (unless (equal? EXIT_SUCCESS (system* #$restic "init"
+                                                 "-r" #$repository))
+               (error "Failed to initialize restic repository: " #$repository))))
          (execlp #$restic #$restic #$@verbose
                  "-r" #$repository
                  #$@extra-flags
-- 
2.45.1

[bug#71639] [PATCH WIP 2/5] services: backup: Add password-command support to restic-service

[Richard Sent]

* gnu/services/backup.scm (restic-backup-job): Add password-command.
(verify-restic-backup-job-configuration): Create.
(restic-backup-job-program): Set either RESTIC_PASSWORD or
RESTIC_PASSWORD_COMMAND depending on what is configured.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0
---
 doc/guix.texi           |  7 +++++++
 gnu/services/backup.scm | 42 ++++++++++++++++++++++++++++++++++++-----
 2 files changed, 44 insertions(+), 5 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 63c9cbd1a7..f22d679023 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41344,6 +41344,13 @@ Miscellaneous Services
 that will be used to set the @env{RESTIC_PASSWORD} environment variable
 for the current job.
 
+@item @code{password-command} (type: file-like)
+String path or file-like object representing the executable file that
+prints password to stdout.  If a file-like object is used, it is placed
+in the store globally executable and in plain text.  The executable
+should be designed such that it does not compromise the password if an
+unauthorized user runs it.
+
 @item @code{schedule} (type: gexp-or-string)
 A string or a gexp that will be passed as time specification in the
 mcron job specification (@pxref{Syntax, mcron job specifications,,
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index eeef11eae7..2471d0ea7b 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -66,6 +66,9 @@ (define (lowerable? value)
 (define list-of-lowerables?
   (list-of lowerable?))
 
+(define-maybe/no-serialization string)
+(define-maybe/no-serialization file-like)
+
 (define-configuration/no-serialization restic-backup-job
   (restic
    (package restic)
@@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job
    (string)
    "The restic repository target of this job.")
   (password-file
-   (string)
+   (maybe-string)
    "Name of the password file, readable by the configured @code{user}, that
 will be used to set the @code{RESTIC_PASSWORD} environment variable for the
 current job.")
+  (password-command
+   (maybe-file-like)
+   "Name of the password command that, when run, returns the password over
+stdin. Due to the nature of the store this command will be globally executable
+and should have external protections to ensure other users cannot retrieve the
+password. This overrides password-file.")
   (schedule
    (gexp-or-string)
    "A string or a gexp that will be passed as time specification in the mcron
@@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job
    "A list of values that are lowered to strings.  These will be passed as
 command-line arguments to the current job @command{restic backup} invokation."))
 
+(define (verify-restic-backup-job-configuration config)
+  (unless (or (maybe-value-set? (restic-backup-job-password-file config))
+              (maybe-value-set? (restic-backup-job-password-command config)))
+    (error "either password-file or password-command must be configured."))
+  (when (and (maybe-value-set? (restic-backup-job-password-file config))
+             (maybe-value-set? (restic-backup-job-password-command config)))
+    (error "password-file and password-command can not be configured simultaneously.")))
+
 (define list-of-restic-backup-jobs?
   (list-of restic-backup-job?))
 
@@ -113,12 +130,22 @@ (define-configuration/no-serialization restic-backup-configuration
    "The list of backup jobs for the current system."))
 
 (define (restic-backup-job-program config)
+  (define (maybe-value-or-false maybe)
+    (if (maybe-value-set? maybe)
+        maybe
+        #f))
+
+  ;; TODO: Find a place to also verify restic-backup-configuration. Mainly that jobs >=1
+  (verify-restic-backup-job-configuration config)
+
   (let ((restic
          (file-append (restic-backup-job-restic config) "/bin/restic"))
         (repository
          (restic-backup-job-repository config))
         (password-file
-         (restic-backup-job-password-file config))
+         (maybe-value-or-false (restic-backup-job-password-file config)))
+        (password-command
+         (maybe-value-or-false (restic-backup-job-password-command config)))
         (files
          (restic-backup-job-files config))
         (extra-flags
@@ -134,9 +161,14 @@ (define (restic-backup-job-program config)
      #~(begin
          (use-modules (ice-9 popen)
                       (ice-9 rdelim))
-         (setenv "RESTIC_PASSWORD"
-                 (with-input-from-file #$password-file read-line))
-
+         (or (and=> #$password-file (lambda (x)
+                                      (setenv "RESTIC_PASSWORD"
+                                              (with-input-from-file x read-line))))
+             (and=> #$password-command (lambda (x)
+                                         (setenv "RESTIC_PASSWORD_COMMAND" x)))
+             ;; Have a backup error message in case
+             ;; verify-restic-backup-job-configuration is messed with
+             (error "Neither password-file or password-command set"))
          (when #$init?
            ;; Check if the repository exists. See
            ;; https://github.com/restic/restic/issues/1690 and
-- 
2.45.1

[bug#71639] [PATCH WIP 3/5] services: backup: Add extra-packages field to restic-backup-job

[Richard Sent]

* gnu/services/backup.scm (restic-backup-job): Create extra-packages.
(restic-guix-wrapper-package): Add the extra-packages field of every job as
inputs.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02
---
 doc/guix.texi           |  4 ++++
 gnu/services/backup.scm | 33 ++++++++++++++++++++-------------
 2 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index f22d679023..32ce0c86b9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41367,6 +41367,10 @@ Miscellaneous Services
 @item @code{verbose?} (default: @code{#f}) (type: boolean)
 Whether to enable verbose output for the current backup job.
 
+@item @code{extra-packages} (default: @code{'()} (type: list-of-packages)
+The list of extra packages needed for restic to run the current backup
+job.
+
 @item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables)
 A list of values that are lowered to strings.  These will be passed as
 command-line arguments to the current job @command{restic backup}
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 2471d0ea7b..a6d8404a5a 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -41,6 +41,7 @@ (define-module (gnu services backup)
             restic-backup-job-files
             restic-backup-job-init?
             restic-backup-job-verbose?
+            restic-backup-job-extra-packages
             restic-backup-job-extra-flags
 
             restic-backup-configuration
@@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job
    (list-of-lowerables '())
    "The list of files or directories to be backed up.  It must be a list of
 values that can be lowered to strings.")
+  (extra-packages
+   (list-of-packages '())
+   "The list of extra packages needed for restic to run this backup job.")
   (verbose?
    (boolean #f)
    "Whether to enable verbose output for the current backup job.")
@@ -239,22 +243,25 @@ (define (restic-backup-job->mcron-job config)
            #:user #$user)))
 
 (define (restic-guix-wrapper-package jobs)
-  (package
-    (name "restic-backup-service-wrapper")
-    (version "0.0.0")
-    (source (restic-guix jobs))
-    (build-system copy-build-system)
-    (arguments
-     (list #:install-plan #~'(("./" "/bin"))))
-    (home-page "https://restic.net")
-    (synopsis
-     "Easily interact from the CLI with Guix configured backups")
-    (description
-     "This package provides a simple wrapper around @code{restic}, handled
+  (let ((extra-packages (append-map restic-backup-job-extra-packages
+                                    jobs)))
+    (package
+      (name "restic-backup-service-wrapper")
+      (version "0.0.0")
+      (source (restic-guix restic-package jobs))
+      (build-system copy-build-system)
+      (arguments
+       (list #:install-plan #~'(("./" "/bin"))))
+      (home-page "https://restic.net")
+      (synopsis
+       "Easily interact from the CLI with Guix configured backups")
+      (description
+       "This package provides a simple wrapper around @code{restic}, handled
 by the @code{restic-backup-service-type}.  It allows for easily interacting
 with Guix configured backup jobs, for example for manually triggering a backup
 without waiting for the scheduled job to run.")
-    (license license:gpl3+)))
+      (inputs extra-packages)
+      (license license:gpl3+))))
 
 (define restic-backup-service-profile
   (lambda (config)
-- 
2.45.1

[bug#71639] [PATCH WIP 4/5] services: backup: Move restic package to restic-configuration

[Richard Sent]

* gnu/services/backup.scm (restic-backup-configuration): Add restic field.
(restic-backup-job-program): Add restic package to function signature.
(restic-guix): Ditto.
(restic-guix-wrapper-package): Ditto. Add restic package as package input.
(restic-backup-service-profile): Remove excess lamba.

* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b
---
 doc/guix.texi           |  6 +++---
 gnu/services/backup.scm | 38 ++++++++++++++++++++------------------
 2 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 32ce0c86b9..e4379c5e1c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41311,6 +41311,9 @@ Miscellaneous Services
 Available @code{restic-backup-configuration} fields are:
 
 @table @asis
+@item @code{restic} (default: @code{restic}) (type: package)
+The restic package to use for all jobs.
+
 @item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs)
 The list of backup jobs for the current system.
 
@@ -41327,9 +41330,6 @@ Miscellaneous Services
 Available @code{restic-backup-job} fields are:
 
 @table @asis
-@item @code{restic} (default: @code{restic}) (type: package)
-The restic package to be used for the current job.
-
 @item @code{user} (default: @code{"root"}) (type: string)
 The user used for running the current job.
 
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index a6d8404a5a..a59cf08c71 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -29,10 +29,10 @@ (define-module (gnu services backup)
   #:use-module (guix modules)
   #:use-module (guix packages)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:export (restic-backup-job
             restic-backup-job?
             restic-backup-job-fields
-            restic-backup-job-restic
             restic-backup-job-user
             restic-backup-job-name
             restic-backup-job-repository
@@ -46,6 +46,7 @@ (define-module (gnu services backup)
 
             restic-backup-configuration
             restic-backup-configuration?
+            restic-backup-configuration-restic
             restic-backup-configuration-fields
             restic-backup-configuration-jobs
 
@@ -71,9 +72,6 @@ (define-maybe/no-serialization string)
 (define-maybe/no-serialization file-like)
 
 (define-configuration/no-serialization restic-backup-job
-  (restic
-   (package restic)
-   "The restic package to be used for the current job.")
   (user
    (string "root")
    "The user used for running the current job.")
@@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs?
   (list-of restic-backup-job?))
 
 (define-configuration/no-serialization restic-backup-configuration
+  (restic
+   (package restic)
+   "The restic package to be used.")
   (jobs
    (list-of-restic-backup-jobs '())
    "The list of backup jobs for the current system."))
 
-(define (restic-backup-job-program config)
+(define (restic-backup-job-program restic-package config)
   (define (maybe-value-or-false maybe)
     (if (maybe-value-set? maybe)
         maybe
@@ -143,7 +144,7 @@ (define (restic-backup-job-program config)
   (verify-restic-backup-job-configuration config)
 
   (let ((restic
-         (file-append (restic-backup-job-restic config) "/bin/restic"))
+         (file-append restic-package "/bin/restic"))
         (repository
          (restic-backup-job-repository config))
         (password-file
@@ -191,7 +192,7 @@ (define (restic-backup-job-program config)
                  #$@extra-flags
                  "backup" #$@files)))))
 
-(define (restic-guix jobs)
+(define (restic-guix restic-package jobs)
   (program-file
    "restic-guix"
    #~(begin
@@ -199,7 +200,7 @@ (define (restic-guix jobs)
                     (srfi srfi-1))
 
        (define names '#$(map restic-backup-job-name jobs))
-       (define programs '#$(map restic-backup-job-program jobs))
+       (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs))
 
        (define (get-program name)
          (define idx
@@ -242,13 +243,13 @@ (define (restic-backup-job->mcron-job config)
            #$(string-append "restic-guix backup " name)
            #:user #$user)))
 
-(define (restic-guix-wrapper-package jobs)
+(define (restic-guix-wrapper-package restic jobs)
   (let ((extra-packages (append-map restic-backup-job-extra-packages
                                     jobs)))
     (package
       (name "restic-backup-service-wrapper")
       (version "0.0.0")
-      (source (restic-guix restic-package jobs))
+      (source (restic-guix restic jobs))
       (build-system copy-build-system)
       (arguments
        (list #:install-plan #~'(("./" "/bin"))))
@@ -260,16 +261,17 @@ (define (restic-guix-wrapper-package jobs)
 by the @code{restic-backup-service-type}.  It allows for easily interacting
 with Guix configured backup jobs, for example for manually triggering a backup
 without waiting for the scheduled job to run.")
-      (inputs extra-packages)
+      (inputs (cons restic extra-packages))
       (license license:gpl3+))))
 
-(define restic-backup-service-profile
-  (lambda (config)
-    (define jobs (restic-backup-configuration-jobs config))
-    (if (> (length jobs) 0)
-        (list
-         (restic-guix-wrapper-package jobs))
-        '())))
+(define (restic-backup-service-profile config)
+  (let ((jobs (restic-backup-configuration-jobs config))
+        (restic (restic-backup-configuration-restic config)))
+    ;; Even if we provide a wrapper we need to add restic to the profile so
+    ;; the service works in containers and VMs which only see a subset of
+    ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to
+    ;; the wrapper package so it is added as an input.
+    (list (restic-guix-wrapper-package restic jobs))))
 
 (define restic-backup-service-type
   (service-type (name 'restic-backup)
-- 
2.45.1

[bug#71639] [PATCH WIP 5/5] tests: Add restic system test.

[Richard Sent]

* gnu/tests/restic.scm: Create.

Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc
---
 gnu/tests/restic.scm | 124 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100644 gnu/tests/restic.scm

diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm
new file mode 100644
index 0000000000..f8ea76538e
--- /dev/null
+++ b/gnu/tests/restic.scm
@@ -0,0 +1,124 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>.
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu tests restic)
+  ;; TODO: Remove what I can from here.
+  #:use-module (gnu bootloader)
+  #:use-module (gnu bootloader grub)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages sync)      ;rclone
+  #:use-module (gnu packages xorg)
+  #:use-module (gnu services)
+  #:use-module (gnu services desktop)
+  #:use-module (gnu services backup)    ;restic
+  #:use-module (gnu services xorg)
+  #:use-module (gnu system)
+  #:use-module (gnu system vm)
+  #:use-module (gnu tests)
+  #:use-module (guix gexp)
+  #:use-module (guix modules)
+  #:use-module (srfi srfi-1)
+  #:export (%test-restic))
+
+(define password "password")
+
+(define password-file
+  (plain-file "password-file" password))
+
+(define password-command
+  (program-file "password-command" #~(display #$password)))
+
+(define (run-restic-test)
+  "Run tests in %restic-os."
+
+  (define os
+    (marionette-operating-system
+     (simple-operating-system (extra-special-file "/root/.restic-test"
+                                                  (plain-file "restic-test"
+                                                              "Hello world!"))
+                              ;; restic-backup-service only takes a string to avoid putting
+                              ;; plaintext entries in the store. Ergo, symlink it.
+                              (extra-special-file "/root/password-file"
+                                                  password-file)
+                              (service restic-backup-service-type
+                                       (restic-backup-configuration
+                                        (jobs
+                                         (list (restic-backup-job
+                                                (name "password-file-backup")
+                                                (repository "/root/restic-password-file-repo")
+                                                (schedule #~'(next-second '(0 15 30 45)))
+                                                (password-file "/root/password-file")
+                                                (files '("/root/.restic-test"))
+                                                (init? #t))
+                                               (restic-backup-job
+                                                (name "password-command-backup")
+                                                (repository "/root/restic-password-command-repo")
+                                                (schedule #~'(next-second '(0 15 30 45)))
+                                                (password-command password-command)
+                                                (files '("/root/.restic-test"))
+                                                (init? #t)))))))
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm (virtual-machine
+              (operating-system os)
+              ;; TODO: We might be able to lower this.
+              (memory-size 1024)))
+
+  (define test
+    (with-imported-modules (source-module-closure
+                            '((gnu build marionette)))
+      #~(begin
+          (use-modules (gnu build marionette)
+                       (srfi srfi-26)
+                       (srfi srfi-64))
+
+          (let ((marionette (make-marionette (list #$vm))))
+
+            (test-runner-current (system-test-runner #$output))
+            (test-begin "restic")
+
+            (test-assert "backup-file-created"
+              (wait-for-file "/root/.restic-test" marionette))
+
+            (test-assert "mcron running"
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (start-service 'mcron))
+               marionette))
+
+            (test-assert "password-file backup completed"
+              (wait-for-file "/root/restic-password-file-repo/config" marionette
+                             ;; Restic takes a second to run, give it a bit
+                             ;; more time.
+                             #:timeout 20))
+
+            (test-assert "password-comand backup completed"
+              (wait-for-file "/root/restic-password-file-repo/config" marionette
+                             #:timeout 20))
+
+            (test-end)))))
+
+  (gexp->derivation "restic-test" test))
+
+(define %test-restic
+  (system-test
+   (name "restic")
+   (description "Basic tests for the restic service.")
+   (value (run-restic-test))))
-- 
2.45.1

[bug#71660] [PATCH v2 0/5] Improve on restic-backup-service

[Richard Sent]

Some minor adjustments to the WIP version, mostly cleanup and removing some
TODOs that snuck in. Thinking on it there's no need to worry about marking
this as WIP. If it winds up merged earlier than expected I can just submit
another patch series.

Richard Sent (5):
  services: backup: Support bootstrapping an initial restic backup
  services: backup: Add password-command support to restic-service
  services: backup: Add extra-packages field to restic-backup-job
  services: backup: Move restic package to restic-configuration
  tests: Add restic system test.

 doc/guix.texi           |  21 ++++++-
 gnu/services/backup.scm | 129 +++++++++++++++++++++++++++++-----------
 gnu/tests/restic.scm    | 119 ++++++++++++++++++++++++++++++++++++
 3 files changed, 231 insertions(+), 38 deletions(-)
 create mode 100644 gnu/tests/restic.scm


base-commit: e32e3d0a03dc17c4c54a91aad053c9036998b601
-- 
2.45.1

[bug#71661] [PATCH v2 1/5] services: backup: Support bootstrapping an initial restic backup

[Richard Sent]

* gnu/services/backup.scm: (restic-backup-job): Add init? field.
(restic-backup-job-program): Initialize repository if init? is set and
repository does not already exist.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d
---
 doc/guix.texi           |  4 ++++
 gnu/services/backup.scm | 19 +++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0102fd0fad..63c9cbd1a7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41353,6 +41353,10 @@ Miscellaneous Services
 The list of files or directories to be backed up.  It must be a list of
 values that can be lowered to strings.
 
+@item @code{init?} (default: @code{#f}) (type: boolean)
+Whether restic-backup-service should check and (if it does not exist)
+initialize the repository before running the backup.
+
 @item @code{verbose?} (default: @code{#f}) (type: boolean)
 Whether to enable verbose output for the current backup job.
 
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 555e9fc959..1279ece88f 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2024 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -38,6 +39,7 @@ (define-module (gnu services backup)
             restic-backup-job-password-file
             restic-backup-job-schedule
             restic-backup-job-files
+            restic-backup-job-init?
             restic-backup-job-verbose?
             restic-backup-job-extra-flags
 
@@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job
   (verbose?
    (boolean #f)
    "Whether to enable verbose output for the current backup job.")
+  (init?
+   (boolean #f)
+   "Whether to attempt to initialize a new repository for automated bootstrap purposes.")
   (extra-flags
    (list-of-lowerables '())
    "A list of values that are lowered to strings.  These will be passed as
@@ -118,6 +123,8 @@ (define (restic-backup-job-program config)
          (restic-backup-job-files config))
         (extra-flags
          (restic-backup-job-extra-flags config))
+        (init?
+         (restic-backup-job-init? config))
         (verbose
          (if (restic-backup-job-verbose? config)
              '("--verbose")
@@ -130,6 +137,18 @@ (define (restic-backup-job-program config)
          (setenv "RESTIC_PASSWORD"
                  (with-input-from-file #$password-file read-line))
 
+         (when #$init?
+           ;; Use cat config to check if the repository exists. See
+           ;; https://github.com/restic/restic/issues/1690 and
+           ;; https://github.com/NixOS/nixpkgs/pull/307962.
+           ;;
+           ;; XXX: restic returns values other than 1 on failure. Check
+           ;; EXIT_SUCCESS instead of EXIT_FAILURE.
+           (unless (or (equal? EXIT_SUCCESS (system* #$restic "cat" "config"
+                                                     "-r" #$repository))
+                       (equal? EXIT_SUCCESS (system* #$restic "init"
+                                                     "-r" #$repository)))
+             (error "Failed to initialize restic repository: " #$repository)))
          (execlp #$restic #$restic #$@verbose
                  "-r" #$repository
                  #$@extra-flags
-- 
2.45.1

[bug#71662] [PATCH v2 2/5] services: backup: Add password-command support to restic-service

[Richard Sent]

* gnu/services/backup.scm (restic-backup-job): Add password-command.
(verify-restic-backup-job-configuration): Create.
(restic-backup-job-program): Set either RESTIC_PASSWORD or
RESTIC_PASSWORD_COMMAND depending on what is configured.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0
---
 doc/guix.texi           |  7 +++++++
 gnu/services/backup.scm | 41 ++++++++++++++++++++++++++++++++++++-----
 2 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 63c9cbd1a7..f22d679023 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41344,6 +41344,13 @@ Miscellaneous Services
 that will be used to set the @env{RESTIC_PASSWORD} environment variable
 for the current job.
 
+@item @code{password-command} (type: file-like)
+String path or file-like object representing the executable file that
+prints password to stdout.  If a file-like object is used, it is placed
+in the store globally executable and in plain text.  The executable
+should be designed such that it does not compromise the password if an
+unauthorized user runs it.
+
 @item @code{schedule} (type: gexp-or-string)
 A string or a gexp that will be passed as time specification in the
 mcron job specification (@pxref{Syntax, mcron job specifications,,
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 1279ece88f..fd904bc9a9 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -66,6 +66,9 @@ (define (lowerable? value)
 (define list-of-lowerables?
   (list-of lowerable?))
 
+(define-maybe/no-serialization string)
+(define-maybe/no-serialization file-like)
+
 (define-configuration/no-serialization restic-backup-job
   (restic
    (package restic)
@@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job
    (string)
    "The restic repository target of this job.")
   (password-file
-   (string)
+   (maybe-string)
    "Name of the password file, readable by the configured @code{user}, that
 will be used to set the @code{RESTIC_PASSWORD} environment variable for the
 current job.")
+  (password-command
+   (maybe-file-like)
+   "An executable file who's path is stored in @code{RESTIC_PASSWORD_COMMAND}.
+When run, the file writes the password to standard output. Due to the nature
+of the store this command will be globally executable and should have external
+protections to ensure unauthorized users cannot retrieve the password.")
   (schedule
    (gexp-or-string)
    "A string or a gexp that will be passed as time specification in the mcron
@@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job
    "A list of values that are lowered to strings.  These will be passed as
 command-line arguments to the current job @command{restic backup} invokation."))
 
+(define (verify-restic-backup-job-configuration config)
+  (unless (or (maybe-value-set? (restic-backup-job-password-file config))
+              (maybe-value-set? (restic-backup-job-password-command config)))
+    (error "either password-file or password-command must be configured."))
+  (when (and (maybe-value-set? (restic-backup-job-password-file config))
+             (maybe-value-set? (restic-backup-job-password-command config)))
+    (error "password-file and password-command can not be configured simultaneously.")))
+
 (define list-of-restic-backup-jobs?
   (list-of restic-backup-job?))
 
@@ -113,12 +130,21 @@ (define-configuration/no-serialization restic-backup-configuration
    "The list of backup jobs for the current system."))
 
 (define (restic-backup-job-program config)
+  (define (maybe-value-or-false maybe)
+    (if (maybe-value-set? maybe)
+        maybe
+        #f))
+
+  (verify-restic-backup-job-configuration config)
+
   (let ((restic
          (file-append (restic-backup-job-restic config) "/bin/restic"))
         (repository
          (restic-backup-job-repository config))
         (password-file
-         (restic-backup-job-password-file config))
+         (maybe-value-or-false (restic-backup-job-password-file config)))
+        (password-command
+         (maybe-value-or-false (restic-backup-job-password-command config)))
         (files
          (restic-backup-job-files config))
         (extra-flags
@@ -134,9 +160,14 @@ (define (restic-backup-job-program config)
      #~(begin
          (use-modules (ice-9 popen)
                       (ice-9 rdelim))
-         (setenv "RESTIC_PASSWORD"
-                 (with-input-from-file #$password-file read-line))
-
+         (or (and=> #$password-file (lambda (x)
+                                      (setenv "RESTIC_PASSWORD"
+                                              (with-input-from-file x read-line))))
+             (and=> #$password-command (lambda (x)
+                                         (setenv "RESTIC_PASSWORD_COMMAND" x)))
+             ;; Have a backup error message in case
+             ;; verify-restic-backup-job-configuration is messed with
+             (error "Neither password-file or password-command set"))
          (when #$init?
            ;; Use cat config to check if the repository exists. See
            ;; https://github.com/restic/restic/issues/1690 and
-- 
2.45.1

[bug#71663] [PATCH v2 3/5] services: backup: Add extra-packages field to restic-backup-job

[Richard Sent]

* gnu/services/backup.scm (restic-backup-job): Create extra-packages.
(restic-guix-wrapper-package): Add the extra-packages field of every job as
inputs.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02
---
 doc/guix.texi           |  4 ++++
 gnu/services/backup.scm | 33 ++++++++++++++++++++-------------
 2 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index f22d679023..32ce0c86b9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41367,6 +41367,10 @@ Miscellaneous Services
 @item @code{verbose?} (default: @code{#f}) (type: boolean)
 Whether to enable verbose output for the current backup job.
 
+@item @code{extra-packages} (default: @code{'()} (type: list-of-packages)
+The list of extra packages needed for restic to run the current backup
+job.
+
 @item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables)
 A list of values that are lowered to strings.  These will be passed as
 command-line arguments to the current job @command{restic backup}
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index fd904bc9a9..bfbacfe590 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -41,6 +41,7 @@ (define-module (gnu services backup)
             restic-backup-job-files
             restic-backup-job-init?
             restic-backup-job-verbose?
+            restic-backup-job-extra-packages
             restic-backup-job-extra-flags
 
             restic-backup-configuration
@@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job
    (list-of-lowerables '())
    "The list of files or directories to be backed up.  It must be a list of
 values that can be lowered to strings.")
+  (extra-packages
+   (list-of-packages '())
+   "The list of extra packages needed for restic to run this backup job.")
   (verbose?
    (boolean #f)
    "Whether to enable verbose output for the current backup job.")
@@ -237,22 +241,25 @@ (define (restic-backup-job->mcron-job config)
            #:user #$user)))
 
 (define (restic-guix-wrapper-package jobs)
-  (package
-    (name "restic-backup-service-wrapper")
-    (version "0.0.0")
-    (source (restic-guix jobs))
-    (build-system copy-build-system)
-    (arguments
-     (list #:install-plan #~'(("./" "/bin"))))
-    (home-page "https://restic.net")
-    (synopsis
-     "Easily interact from the CLI with Guix configured backups")
-    (description
-     "This package provides a simple wrapper around @code{restic}, handled
+  (let ((extra-packages (append-map restic-backup-job-extra-packages
+                                    jobs)))
+    (package
+      (name "restic-backup-service-wrapper")
+      (version "0.0.0")
+      (source (restic-guix restic-package jobs))
+      (build-system copy-build-system)
+      (arguments
+       (list #:install-plan #~'(("./" "/bin"))))
+      (home-page "https://restic.net")
+      (synopsis
+       "Easily interact from the CLI with Guix configured backups")
+      (description
+       "This package provides a simple wrapper around @code{restic}, handled
 by the @code{restic-backup-service-type}.  It allows for easily interacting
 with Guix configured backup jobs, for example for manually triggering a backup
 without waiting for the scheduled job to run.")
-    (license license:gpl3+)))
+      (inputs extra-packages)
+      (license license:gpl3+))))
 
 (define restic-backup-service-profile
   (lambda (config)
-- 
2.45.1

[bug#71639] [PATCH v2 4/5] services: backup: Move restic package to restic-configuration

[Richard Sent]

* gnu/services/backup.scm (restic-backup-configuration): Add restic field.
(restic-backup-job-program): Add restic package to function signature.
(restic-guix): Ditto.
(restic-guix-wrapper-package): Ditto. Add restic package as package input.
(restic-backup-service-profile): Remove excess lamba.

* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b
---
 doc/guix.texi           |  6 +++---
 gnu/services/backup.scm | 38 ++++++++++++++++++++------------------
 2 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 32ce0c86b9..e4379c5e1c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41311,6 +41311,9 @@ Miscellaneous Services
 Available @code{restic-backup-configuration} fields are:
 
 @table @asis
+@item @code{restic} (default: @code{restic}) (type: package)
+The restic package to use for all jobs.
+
 @item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs)
 The list of backup jobs for the current system.
 
@@ -41327,9 +41330,6 @@ Miscellaneous Services
 Available @code{restic-backup-job} fields are:
 
 @table @asis
-@item @code{restic} (default: @code{restic}) (type: package)
-The restic package to be used for the current job.
-
 @item @code{user} (default: @code{"root"}) (type: string)
 The user used for running the current job.
 
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index bfbacfe590..5e2add088e 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -29,10 +29,10 @@ (define-module (gnu services backup)
   #:use-module (guix modules)
   #:use-module (guix packages)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:export (restic-backup-job
             restic-backup-job?
             restic-backup-job-fields
-            restic-backup-job-restic
             restic-backup-job-user
             restic-backup-job-name
             restic-backup-job-repository
@@ -46,6 +46,7 @@ (define-module (gnu services backup)
 
             restic-backup-configuration
             restic-backup-configuration?
+            restic-backup-configuration-restic
             restic-backup-configuration-fields
             restic-backup-configuration-jobs
 
@@ -71,9 +72,6 @@ (define-maybe/no-serialization string)
 (define-maybe/no-serialization file-like)
 
 (define-configuration/no-serialization restic-backup-job
-  (restic
-   (package restic)
-   "The restic package to be used for the current job.")
   (user
    (string "root")
    "The user used for running the current job.")
@@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs?
   (list-of restic-backup-job?))
 
 (define-configuration/no-serialization restic-backup-configuration
+  (restic
+   (package restic)
+   "The restic package to be used.")
   (jobs
    (list-of-restic-backup-jobs '())
    "The list of backup jobs for the current system."))
 
-(define (restic-backup-job-program config)
+(define (restic-backup-job-program restic-package config)
   (define (maybe-value-or-false maybe)
     (if (maybe-value-set? maybe)
         maybe
@@ -142,7 +143,7 @@ (define (restic-backup-job-program config)
   (verify-restic-backup-job-configuration config)
 
   (let ((restic
-         (file-append (restic-backup-job-restic config) "/bin/restic"))
+         (file-append restic-package "/bin/restic"))
         (repository
          (restic-backup-job-repository config))
         (password-file
@@ -189,7 +190,7 @@ (define (restic-backup-job-program config)
                  #$@extra-flags
                  "backup" #$@files)))))
 
-(define (restic-guix jobs)
+(define (restic-guix restic-package jobs)
   (program-file
    "restic-guix"
    #~(begin
@@ -197,7 +198,7 @@ (define (restic-guix jobs)
                     (srfi srfi-1))
 
        (define names '#$(map restic-backup-job-name jobs))
-       (define programs '#$(map restic-backup-job-program jobs))
+       (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs))
 
        (define (get-program name)
          (define idx
@@ -240,13 +241,13 @@ (define (restic-backup-job->mcron-job config)
            #$(string-append "restic-guix backup " name)
            #:user #$user)))
 
-(define (restic-guix-wrapper-package jobs)
+(define (restic-guix-wrapper-package restic jobs)
   (let ((extra-packages (append-map restic-backup-job-extra-packages
                                     jobs)))
     (package
       (name "restic-backup-service-wrapper")
       (version "0.0.0")
-      (source (restic-guix restic-package jobs))
+      (source (restic-guix restic jobs))
       (build-system copy-build-system)
       (arguments
        (list #:install-plan #~'(("./" "/bin"))))
@@ -258,16 +259,17 @@ (define (restic-guix-wrapper-package jobs)
 by the @code{restic-backup-service-type}.  It allows for easily interacting
 with Guix configured backup jobs, for example for manually triggering a backup
 without waiting for the scheduled job to run.")
-      (inputs extra-packages)
+      (inputs (cons restic extra-packages))
       (license license:gpl3+))))
 
-(define restic-backup-service-profile
-  (lambda (config)
-    (define jobs (restic-backup-configuration-jobs config))
-    (if (> (length jobs) 0)
-        (list
-         (restic-guix-wrapper-package jobs))
-        '())))
+(define (restic-backup-service-profile config)
+  (let ((jobs (restic-backup-configuration-jobs config))
+        (restic (restic-backup-configuration-restic config)))
+    ;; Even if we provide a wrapper we need to add restic to the profile so
+    ;; the service works in containers and VMs which only see a subset of
+    ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to
+    ;; the wrapper package so it is added as an input.
+    (list (restic-guix-wrapper-package restic jobs))))
 
 (define restic-backup-service-type
   (service-type (name 'restic-backup)
-- 
2.45.1

[bug#71665] [PATCH v2 5/5] tests: Add restic system test.

[Richard Sent]

* gnu/tests/restic.scm: Create.

Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc
---
 gnu/tests/restic.scm | 119 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)
 create mode 100644 gnu/tests/restic.scm

diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm
new file mode 100644
index 0000000000..8d29ff441b
--- /dev/null
+++ b/gnu/tests/restic.scm
@@ -0,0 +1,119 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>.
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu tests restic)
+  #:use-module (gnu bootloader)
+  #:use-module (gnu bootloader grub)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages sync)      ;rclone
+  #:use-module (gnu services)
+  #:use-module (gnu services backup)    ;restic
+  #:use-module (gnu system)
+  #:use-module (gnu system vm)
+  #:use-module (gnu tests)
+  #:use-module (guix gexp)
+  #:use-module (guix modules)
+  #:use-module (srfi srfi-1)
+  #:export (%test-restic))
+
+(define password "password")
+
+(define password-file
+  (plain-file "password-file" password))
+
+(define password-command
+  (program-file "password-command" #~(display #$password)))
+
+(define (run-restic-test)
+  "Run tests in %restic-os."
+
+  (define os
+    (marionette-operating-system
+     (simple-operating-system (extra-special-file "/root/.restic-test"
+                                                  (plain-file "restic-test"
+                                                              "Hello world!"))
+                              ;; restic-backup-service only takes a string to avoid putting
+                              ;; plaintext entries in the store. Ergo, symlink it.
+                              (extra-special-file "/root/password-file"
+                                                  password-file)
+                              (service restic-backup-service-type
+                                       (restic-backup-configuration
+                                        (jobs
+                                         (list (restic-backup-job
+                                                (name "password-file-backup")
+                                                (repository "/root/restic-password-file-repo")
+                                                (schedule #~'(next-second '(0 15 30 45)))
+                                                (password-file "/root/password-file")
+                                                (files '("/root/.restic-test"))
+                                                (init? #t))
+                                               (restic-backup-job
+                                                (name "password-command-backup")
+                                                (repository "/root/restic-password-command-repo")
+                                                (schedule #~'(next-second '(0 15 30 45)))
+                                                (password-command password-command)
+                                                (files '("/root/.restic-test"))
+                                                (init? #t)))))))
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm (virtual-machine
+              (operating-system os)
+              (memory-size 512)))
+
+  (define test
+    (with-imported-modules (source-module-closure
+                            '((gnu build marionette)))
+      #~(begin
+          (use-modules (gnu build marionette)
+                       (srfi srfi-26)
+                       (srfi srfi-64))
+
+          (let ((marionette (make-marionette (list #$vm))))
+
+            (test-runner-current (system-test-runner #$output))
+            (test-begin "restic")
+
+            (test-assert "backup-file-created"
+              (wait-for-file "/root/.restic-test" marionette))
+
+            (test-assert "mcron running"
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (start-service 'mcron))
+               marionette))
+
+            (test-assert "password-file backup completed"
+              (wait-for-file "/root/restic-password-file-repo/config" marionette
+                             ;; Restic takes a second to run, give it a bit
+                             ;; more time.
+                             #:timeout 20))
+
+            (test-assert "password-comand backup completed"
+              (wait-for-file "/root/restic-password-file-repo/config" marionette
+                             #:timeout 20))
+
+            (test-end)))))
+
+  (gexp->derivation "restic-test" test))
+
+(define %test-restic
+  (system-test
+   (name "restic")
+   (description "Basic tests for the restic service.")
+   (value (run-restic-test))))
-- 
2.45.1

[bug#71639] [PATCHv2 0/5] Improve on restic-backup-service

[paul via Guix-patches via]

Hi Richard,

thank you for this work, it looks really useful and in general I think 
the patches are ok but I have some points I'd like to discuss.

I think it would be nice to have an init action for the restic-guix 
command shipped with the service and move there the logic for 
initializing repositories. The service then could be adapted in a way 
that it would call the restic-guix init before calling restic-guix 
backup. Would you be interested in implementing this?

At last, my use case for having a restic package field for each job is 
to have some critical jobs that I don't want to touch running with 
Guix's bootstrapped restic package and some personal jobs that I run 
with a restic 0.16 binary package I have in my personal channel . I'm 
not sure this warrants a field in each single job, but they are optional 
anyway. Anyway I wouldn't consider this a blocker and if the Guix 
project has some guidelines in this sense I'd say follow them.

Cheers,

giacomo

[bug#71639] [PATCHv2 0/5] Improve on restic-backup-service

[Richard Sent]

Hi giacomo,

> I think it would be nice to have an init action for the restic-guix
> command shipped with the service and move there the logic for
> initializing repositories. The service then could be adapted in a way
> that it would call the restic-guix init before calling restic-guix
> backup. Would you be interested in implementing this?

I'll take a look at it and see what the code looks like. It might be a
bit of effort to get that working cleanly while avoiding code dupe. For
instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup
program actions.

Where do you think the appropriate place to check init? and run the init
action is if it's no longer encapsulated in the backup action? A
conditional in restic-backup-job->mcron-job before launching restic-guix
backup? A one-shot shepherd service?

The former will necessitate adding a job string to display when running
$ herd schedule mcron. A downside of the latter is if the restic
repository is deleted for one reason or another that backup job will
fail until the system is rebooted, which isn't immediately obvious.
Personally I prefer the mcron-job conditional.

> At last, my use case for having a restic package field for each job is 
> to have some critical jobs that I don't want to touch running with 
> Guix's bootstrapped restic package and some personal jobs that I run 
> with a restic 0.16 binary package I have in my personal channel . I'm 
> not sure this warrants a field in each single job, but they are optional 
> anyway. Anyway I wouldn't consider this a blocker and if the Guix 
> project has some guidelines in this sense I'd say follow them.

Ah, I see. To me this is where a per-job restic-override or similarly
named field makes sense. This way we can have a default "global" restic
package configured at a service level while still allowing individual
jobs to use a custom restic package. I imagine this restic-override
field would be a "maybe-file-like" either set to #f or a custom restic
package.

Thanks for the feedback!

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

[bug#71639] [PATCHv2 0/5] Improve on restic-backup-service

[paul via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Hi Richard,

On 6/27/24 05:56, Richard Sent wrote:
> I'll take a look at it and see what the code looks like. It might be a
> bit of effort to get that working cleanly while avoiding code dupe. For
> instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup
> program actions.
Yes, 100% agree. What I would do is (and probably there is a better way):

- rename restic-backup-job-program to restic-action-job-program or 
something similar. the procedure would be the same but it would take two 
additional arguments: an action-name argument or similar that would 
replace all instances of "backup" and I think this new procedure should 
not access the files field of the job record but it instead it would 
take an action-arguments argument which would replace #$@files with 
#$@action-arguments in the execlp call. This way, I hope, it could be 
general enough to support additional actions besides init in the future, 
such as purge or restore.

- create a new restic-backup-job-program and a restic-init-job-program 
or similar procedures that would call the new generic 
restic-action-job-program

- inside the restic-guix procedure I'd rename the programs list to 
backup-programs and I would create a similar list for the init action 
with #f placeholders for jobs that do not require initializing, and I 
would check the conditional inside the backup procedure something like:

(define (get-program name programs)
   [...])

(define (init args)
  [...])

(define (backup args)
   (define name (third args))
   (define backup-program (get-program name backup-programs))
   (define init-program (get-program name init-programs))
   (when init-program
     (init args))
   (execlp backup-program backup-program))


this requires the init action not to exec the script resulting from the 
gexp but I can't think of a reason why it could be problematic right now 
so I guess it is ok.

Please take all of the above as a proposal there are for sure things to 
make better.

> Thanks for the feedback!

Thank you very much for your work,

giacomo

[-- Attachment #2: Type: text/html, Size: 2729 bytes --]

[bug#71639] [PATCHv2 0/5] Improve on restic-backup-service

[Fabio Natali via Guix-patches via]

Hi Giacomo,

Thanks for adding the Restic Service to Guix.

Hi Richard,

Thanks for patch request 71639, which includes a few changes I'm looking
forward to. For instance, I like the idea of exposing the
'RESTIC_PASSWORD_COMMAND' variable directly via the service
configuration.

I think I might still be able to use the 'extra-flags' field to specify
a '--password-command' - if I understand it correctly - but direct
access to the option would be neat indeed.

Anything I can do to help with the patch? Could it make sense to break
the patch down into a couple of stages, possibly leaving the most
impactful changes for a later stage?

Thanks, cheers, Fabio.

[bug#71639] [PATCHv2 0/5] Improve on restic-backup-service

[paul via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

Dear Fabio and Richard,

I apologize for blocking this issue with my proposal. I think Fabio's 
proposal of splitting the changes in this patch is the best way forward, 
if you agree as well Richard. I submitted [0] to refactor the 
restic-guix procedure in a way that it can support many different commands.

After it gets in it should be sufficient to add "init" to 
%restic-guix-supported-actions to have a working restic-guix init 
invokation. it should be then matter of understanding where it is better 
to put it. Lately I was thinking that may be best to have initialization 
as a one shot Shepherd service that check whether a given job is 
supposed to have its repository initialized and if that's the case it 
could run restic-guix init name-of the job. Please Richard let me know 
what you think of this approach and whether you would still be 
interested in implementing it, thank you very much!

I think the following subdivision should match all requirements we 
stated until now:

1. Improve the restic backup system service . This can be done in the 
current issue #71639:

services: backup: Add password-command support to restic-service
services: backup: Add extra-packages field to restic-backup-job
services: backup: Move restic package to restic-configuration

2. Refactor the restic-guix function to allow for more restic commands 
to be wrapped. This can be done in issue #72803:

services: restic-backup: Add more restic commands to the restic-guix 
package.

3. Allow for repositories to be initialized with a restic-guix init 
command. This commit could be adapted and moved to a new branch based on 
#72803:

services: backup: Support bootstrapping an initial restic backup

What do you think? Could this be a suitable action plan?

Thank you very much for your work,

giacomo


[0]: https://issues.guix.gnu.org/72803

[-- Attachment #2: Type: text/html, Size: 2546 bytes --]

[bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service

[Fabio Natali via Guix-patches via]

> I apologize for blocking this issue with my proposal. I think Fabio's
> proposal of splitting the changes in this patch is the best way
> forward, if you agree as well Richard. I submitted [0] to refactor the
> restic-guix procedure in a way that it can support many different
> commands.

Hi Giacomo,

Thanks for this!

> 1. Improve the restic backup system service [...]

> 2. Refactor the restic-guix function [...]

> 3. Allow for repositories to be initialized with a restic-guix init [...]

I think this is a perfectly sensible plan. For what it's worth, I've
provided some little feedback around point 2 as part of #72803.

As a further point, I also think we should manage to update Restic
itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
2019). With such an out-of-date package, the service also risks to loose
a little bit of its value.

I've just sent a quick follow-up to #63019 in the hope that we might be
able to get this sorted soon.

Thanks, cheers, Fabio.

[bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service

[Richard Sent]

Hi Giacomo and Fabio!

I've been struck by the the law of development: the backlog will always
expand to exceed the time available.

>> I apologize for blocking this issue with my proposal. I think Fabio's
>> proposal of splitting the changes in this patch is the best way
>> forward, if you agree as well Richard. I submitted [0] to refactor the
>> restic-guix procedure in a way that it can support many different
>> commands.

That sounds fine with me. I'll try to rebase and resubmit as time is
available (hopefully the next few days......). If my schedule is
inconvenient feel free to do with my patches what you will.

>> 1. Improve the restic backup system service [...]
>> 2. Refactor the restic-guix function [...]
>> 3. Allow for repositories to be initialized with a restic-guix init
>> [...]

Ideally, we'd also move the system tests to 1 so further work can be
trivially verified by running $ make check-system TESTS=restic. That'll
require writing an intermediary bootstrapping function in the test
suite.

> I think this is a perfectly sensible plan. For what it's worth, I've
> provided some little feedback around point 2 as part of #72803.
>
> As a further point, I also think we should manage to update Restic
> itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
> 2019). With such an out-of-date package, the service also risks to loose
> a little bit of its value.

That would be lovely! One particular feature I'm hopeful for is
compression support.

Unfortunately restic stopped providing a vendor directory in their
tarball after 0.9.6 and seemingly has no interest in providing it again
[1]. Last time I tried guix import with restic I saw some odd behavior.
Someone with more experience in Go may have better luck.

>> Lately I was thinking that may be best to have initialization as a
>> one shot Shepherd service that check whether a given job is supposed
>> to have its repository initialized and if that's the case it could
>> run restic-guix init name-of the job.

Is the intent to have one "mega" one-shot service that checks every job,
or a unique one-shot service per job? I greatly prefer the former as $
sudo herd status is crowded as it is.

It's worth noting users will need the ability to specify custom Shepherd
requirements so e.g. backup initialization to a remote file-share isn't
triggered until the file-system-/my/remote/nas symbol is provisioned by
Shepherd.

On an unrelated note, Shepherd timers are interesting and may allow for
better user management/error signaling instead of mcron. I'm not
proposing we refactor to use them now (especially since they're not even
in stable), but just thought I'd mention them. [2]

[1]: https://github.com/restic/restic/issues/3945
[2]: https://lists.gnu.org/archive/html/guix-devel/2024-03/msg00247.html

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

[bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service

[Fabio Natali via Guix-patches via]

Hi Richard and Giacomo.

> >> 1. Improve the restic backup system service [...]
[...]
> Ideally, we'd also move the system tests to 1 so further work can be
> trivially verified by running $ make check-system
> TESTS=restic. That'll require writing an intermediary bootstrapping
> function in the test suite.

+1

> > As a further point, I also think we should manage to update Restic
> > itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
> > 2019). With such an out-of-date package, the service also risks to loose
> > a little bit of its value.
>
> That would be lovely! One particular feature I'm hopeful for is
> compression support.
>
> Unfortunately restic stopped providing a vendor directory in their
> tarball after 0.9.6 and seemingly has no interest in providing it
> again [1]. Last time I tried guix import with restic I saw some odd
> behavior. Someone with more experience in Go may have better luck.

> [1]: https://github.com/restic/restic/issues/3945

Thanks for the link to the Restic's issue tracker. I think it's actually
better to have the dependencies unbundled and built/imported
individually - although this is causing us some pain at the moment.

Some of the 'guix import go --recursive restic' errors seem to be caused
by Go projects using a slightly different subfolder scheme? There seem
to be various open issues (and semi-finalised patches) around this, such
as:

- https://issues.guix.gnu.org/issue/52362
- https://issues.guix.gnu.org/issue/63001
- https://issues.guix.gnu.org/issue/63647
- https://issues.guix.gnu.org/issue/64035
- https://issues.guix.gnu.org/issue/64036

I'll follow up on one of the threads above and report back here if I
find anything useful.

Thanks, cheers, Fabio.