[bug#74151] [PATCH 0/7] Add anonip system test.

[bug#74151] [PATCH 0/7] Add anonip system test.

[Maxim Cournoyer]

This was developed as part of investigating the source of bug #59181 ([berlin]
web services fail to start on reboot due to anonip).

There is no issue having anonip run and the system reboot (as should be
expected as /var/run is supposed to be cleared thus the named pipes should be
recreated every boot).  This points to a race with nginx that would "win"
creating its log files under /var/run/anonip/* before the anonip service does.

It'll need further investigating in that direction.

Maxim Cournoyer (7):
  doc: Use @table @code for anonip-configuration doc.
  services: anonip: Add 'debug?' configuration field.
  system/vm: Fix virtual-machine bug.
  tests: web: Have the retry-on-error throw on exhausted attempts.
  services: herd: Export 'eval-there' in API.
  build: marionette: Make it possible to reboot VM during tests.
  tests: Add anonip system test.

 doc/guix.texi            |  21 ++++---
 gnu/build/marionette.scm |   2 +-
 gnu/services/herd.scm    |   1 +
 gnu/services/web.scm     |   8 ++-
 gnu/system/vm.scm        |   9 ++-
 gnu/tests/web.scm        | 117 ++++++++++++++++++++++++++++++++++++++-
 6 files changed, 142 insertions(+), 16 deletions(-)


base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
-- 
2.46.0

[bug#74151] [PATCH 1/7] doc: Use @table @code for anonip-configuration doc.

[Maxim Cournoyer]

* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
 doc/guix.texi | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
 
 The following optional settings may be provided:
 
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
-@item @code{column}
+@item column
 A 1-based indexed column number.  Assume IP address is in the specified
 column (default is 1).
 
-@item @code{replacement}
+@item replacement
 Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
 
-@item @code{ipv4mask}
+@item ipv4mask
 Number of bits to mask in IPv4 addresses.
 
-@item @code{ipv6mask}
+@item ipv6mask
 Number of bits to mask in IPv6 addresses.
 
-@item @code{increment}
+@item increment
 Increment the IP address by the given number.  By default this is zero.
 
-@item @code{delimiter}
+@item delimiter
 Log delimiter string.
 
-@item @code{regex}
+@item regex
 Regular expression for detecting IP addresses.  Use this instead of @code{column}.
 @end table
 @end deftp
-- 
2.46.0

[bug#74151] [PATCH 2/7] services: anonip: Add 'debug?' configuration field.

[Maxim Cournoyer]

* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
 doc/guix.texi        | 3 +++
 gnu/services/web.scm | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
 The following optional settings may be provided:
 
 @table @code
+@item debug?
+Print debug messages when @code{#true}.
+
 @item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
             anonip-configuration-anonip
             anonip-configuration-input
             anonip-configuration-output
+            anonip-configuration-debug?
             anonip-configuration-skip-private?
             anonip-configuration-column
             anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
                      (default anonip))
   (input             anonip-configuration-input)       ;string
   (output            anonip-configuration-output)      ;string
+  (debug?            anonip-configuration-debug?       ;boolean
+                     (default #f))
   (skip-private?     anonip-configuration-skip-private? ;boolean
                      (default #f))
   (column            anonip-configuration-column       ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
                                     "/bin/anonip")
                      (string-append "--input=" #$input)
                      (string-append "--output=" #$output))
+               (if #$(anonip-configuration-debug? config)
+                   '("--debug") (list))
                (if #$(anonip-configuration-skip-private? config)
                    '("--skip-private") (list))
                '#$(optional anonip-configuration-column "--column")
                '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
                '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
                '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement
-                            "--replacement")
+               '#$(optional anonip-configuration-replacement "--replacement")
                '#$(optional anonip-configuration-delimiter "--delimiter")
                '#$(optional anonip-configuration-regex "--regex"))
               ;; Run in a UTF-8 locale
-- 
2.46.0

[bug#74151] [PATCH 3/7] system/vm: Fix virtual-machine bug.

[Maxim Cournoyer]

The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
 gnu/system/vm.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
                     (default #f)))
 
 (define-syntax virtual-machine
-  (syntax-rules ()
+  (syntax-rules (operating-system)
     "Declare a virtual machine running the specified OS, with the given
 options."
-    ((_ os)                                       ;shortcut
+    ((_ (operating-system os))
+     ;; Also accept the long form (virtual-machine (operating-system os)), for
+     ;; correctness.
+     (%virtual-machine (operating-system os)))
+    ((_ os)                             ;shortcut
      (%virtual-machine (operating-system os)))
     ((_ fields ...)
      (%virtual-machine fields ...))))
-- 
2.46.0

[bug#74151] [PATCH 4/7] tests: web: Have the retry-on-error throw on exhausted attempts.

[Maxim Cournoyer]

Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
 gnu/tests/web.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
            return-value)
           ((#f . error-args)
            (if (>= attempt times)
-               error-args
+               (apply throw error-args)
                (begin
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
-- 
2.46.0

[bug#74151] [PATCH 5/7] services: herd: Export 'eval-there' in API.

[Maxim Cournoyer]

* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
 gnu/services/herd.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
             with-shepherd-action
             current-service
             current-services
+            eval-there
             unload-services
             unload-service
             load-services
-- 
2.46.0

[bug#74151] [PATCH 6/7] build: marionette: Make it possible to reboot VM during tests.

[Maxim Cournoyer]

* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
 gnu/build/marionette.scm | 2 +-
 gnu/system/vm.scm        | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
     (list "-nographic"
           "-monitor" (string-append "unix:" socket-directory "/monitor")
           "-chardev" (string-append "socket,id=repl,path=" socket-directory
-                                    "/repl")
+                                    "/repl,reconnect=1")
           "-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
                                     socket-directory "/qemu-ga")
 
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
             '("-enable-kvm")
             '())
 
-     "-no-reboot"
      "-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
      "-device" "virtio-rng-pci,rng=guix-vm-rng"
 
-- 
2.46.0

[bug#74151] [PATCH 7/7] tests: Add anonip system test.

[Maxim Cournoyer]

* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
 gnu/tests/web.scm | 115 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..f7a5659dc5 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2017, 2018 Clément Lassieur <clement@lassieur.org>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
 ;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
   #:use-module (gnu services networking)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services mail)
+  #:use-module (gnu packages base)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
             %test-php-fpm
             %test-hpcguix-web
             %test-tailon
+            %test-anonip
             %test-patchwork
             %test-agate))
 
@@ -509,6 +512,118 @@ (define %test-tailon
    (description "Connect to a running Tailon server.")
    (value (run-tailon-test))))
 
+\f
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+  ;; Operating system under test.
+  (simple-operating-system
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/http.access.log")
+             (output "/var/log/anonip/http.access.log")
+             (debug? #t)))))
+
+(define (run-anonip-test)
+  (define os
+    (marionette-operating-system
+     %anonip-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     ;; We are interested in verifying if anonip still launches following a
+     ;; reboot; thus make the base image writable.
+     (volatile? #f)))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (srfi srfi-64)
+                       (gnu build marionette))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "anonip")
+
+          (test-assert "service is running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+             marionette))
+
+          (test-assert "service can be restarted"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (restart-service 'anonip-/var/log/anonip/http.access.log)
+                (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+             marionette))
+
+          (test-assert "ip addresses are anonymized"
+            (marionette-eval
+             '(begin
+                (use-modules (ice-9 textual-ports))
+                (call-with-output-file "/var/run/anonip/http.access.log"
+                  (lambda (port)
+                    (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+                    (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+                (#$retry-on-error
+                 (lambda ()
+                   (call-with-input-file "/var/log/anonip/http.access.log"
+                     (lambda (port)
+                       (let ((content (get-string-all port)))
+                         ;; The expected values are taken from anonip's test
+                         ;; suite (see its test_module.py file).
+                         (or (and (string-contains content "192.168.96.0")
+                                  (string-contains content "2001:db8:85a0::"))
+                             (error "could not find expected anonymized IPs"
+                                    content))))))
+                 #:times 20
+                 #:delay 1))
+             marionette))
+
+          (test-assert "service is running after reboot"
+            (begin
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (eval-there '(begin
+                                 (use-modules (shepherd system))
+                                 (sync) ;ensure the log is fully written
+                                 (reboot))))
+               marionette)
+              ;; Note: a distinct marionette-eval call is needed here; if
+              ;; included in the previous one issuing the reboot,
+              ;; 'wait-for-service' would apparently run before the system had
+              ;; rebooted (and succeed), which would defeat the test.
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+               marionette)))
+
+          (test-end))))
+
+  (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+  (system-test
+   (name "anonip")
+   (description "Anonymize logs via Anonip")
+   (value (run-anonip-test))))
+
 \f
 ;;;
 ;;; Patchwork
-- 
2.46.0

[bug#74151] [PATCH v2 1/9] doc: Use @table @code for anonip-configuration doc.

[Maxim Cournoyer]

* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
 doc/guix.texi | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
 
 The following optional settings may be provided:
 
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
-@item @code{column}
+@item column
 A 1-based indexed column number.  Assume IP address is in the specified
 column (default is 1).
 
-@item @code{replacement}
+@item replacement
 Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
 
-@item @code{ipv4mask}
+@item ipv4mask
 Number of bits to mask in IPv4 addresses.
 
-@item @code{ipv6mask}
+@item ipv6mask
 Number of bits to mask in IPv6 addresses.
 
-@item @code{increment}
+@item increment
 Increment the IP address by the given number.  By default this is zero.
 
-@item @code{delimiter}
+@item delimiter
 Log delimiter string.
 
-@item @code{regex}
+@item regex
 Regular expression for detecting IP addresses.  Use this instead of @code{column}.
 @end table
 @end deftp

base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
-- 
2.46.0

[bug#74151] [PATCH v2 2/9] services: anonip: Add 'debug?' configuration field.

[Maxim Cournoyer]

* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
 doc/guix.texi        | 3 +++
 gnu/services/web.scm | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
 The following optional settings may be provided:
 
 @table @code
+@item debug?
+Print debug messages when @code{#true}.
+
 @item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
             anonip-configuration-anonip
             anonip-configuration-input
             anonip-configuration-output
+            anonip-configuration-debug?
             anonip-configuration-skip-private?
             anonip-configuration-column
             anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
                      (default anonip))
   (input             anonip-configuration-input)       ;string
   (output            anonip-configuration-output)      ;string
+  (debug?            anonip-configuration-debug?       ;boolean
+                     (default #f))
   (skip-private?     anonip-configuration-skip-private? ;boolean
                      (default #f))
   (column            anonip-configuration-column       ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
                                     "/bin/anonip")
                      (string-append "--input=" #$input)
                      (string-append "--output=" #$output))
+               (if #$(anonip-configuration-debug? config)
+                   '("--debug") (list))
                (if #$(anonip-configuration-skip-private? config)
                    '("--skip-private") (list))
                '#$(optional anonip-configuration-column "--column")
                '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
                '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
                '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement
-                            "--replacement")
+               '#$(optional anonip-configuration-replacement "--replacement")
                '#$(optional anonip-configuration-delimiter "--delimiter")
                '#$(optional anonip-configuration-regex "--regex"))
               ;; Run in a UTF-8 locale
-- 
2.46.0

[bug#74151] [PATCH v2 3/9] system/vm: Fix virtual-machine bug.

[Maxim Cournoyer]

The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
 gnu/system/vm.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
                     (default #f)))
 
 (define-syntax virtual-machine
-  (syntax-rules ()
+  (syntax-rules (operating-system)
     "Declare a virtual machine running the specified OS, with the given
 options."
-    ((_ os)                                       ;shortcut
+    ((_ (operating-system os))
+     ;; Also accept the long form (virtual-machine (operating-system os)), for
+     ;; correctness.
+     (%virtual-machine (operating-system os)))
+    ((_ os)                             ;shortcut
      (%virtual-machine (operating-system os)))
     ((_ fields ...)
      (%virtual-machine fields ...))))
-- 
2.46.0

[bug#74151] [PATCH v2 4/9] tests: web: Have the retry-on-error throw on exhausted attempts.

[Maxim Cournoyer]

Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
 gnu/tests/web.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
            return-value)
           ((#f . error-args)
            (if (>= attempt times)
-               error-args
+               (apply throw error-args)
                (begin
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
-- 
2.46.0

[bug#74151] [PATCH v2 5/9] services: herd: Export 'eval-there' in API.

[Maxim Cournoyer]

* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
 gnu/services/herd.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
             with-shepherd-action
             current-service
             current-services
+            eval-there
             unload-services
             unload-service
             load-services
-- 
2.46.0

[bug#74151] [PATCH v2 6/9] build: marionette: Make it possible to reboot VM during tests.

[Maxim Cournoyer]

* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
 gnu/build/marionette.scm | 2 +-
 gnu/system/vm.scm        | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
     (list "-nographic"
           "-monitor" (string-append "unix:" socket-directory "/monitor")
           "-chardev" (string-append "socket,id=repl,path=" socket-directory
-                                    "/repl")
+                                    "/repl,reconnect=1")
           "-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
                                     socket-directory "/qemu-ga")
 
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
             '("-enable-kvm")
             '())
 
-     "-no-reboot"
      "-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
      "-device" "virtio-rng-pci,rng=guix-vm-rng"
 
-- 
2.46.0

[bug#74151] [PATCH v2 6/9] build: marionette: Make it possible to reboot VM during tests.

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> * gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
> parameter.
> * gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.
>
> Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73

[...]

> +++ b/gnu/system/vm.scm
> @@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
>              '("-enable-kvm")
>              '())
>  
> -     "-no-reboot"

I believe the reason we had ‘-no-reboot’ is because otherwise QEMU would
restart the VM when we halt it (?).  I’m not sure actually what the
exact scenario was.

Anyway, if you confirm that other system tests are unaffected by this
change, this is probably good.

Ludo’.

[bug#74151] [PATCH v2 7/9] tests: Add anonip system test.

[Maxim Cournoyer]

* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
 gnu/tests/web.scm | 122 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..6ef32138ed 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2017, 2018 Clément Lassieur <clement@lassieur.org>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
 ;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
   #:use-module (gnu services networking)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services mail)
+  #:use-module (gnu packages base)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
             %test-php-fpm
             %test-hpcguix-web
             %test-tailon
+            %test-anonip
             %test-patchwork
             %test-agate))
 
@@ -509,6 +512,125 @@ (define %test-tailon
    (description "Connect to a running Tailon server.")
    (value (run-tailon-test))))
 
+\f
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+  ;; Operating system under test.
+  (simple-operating-system
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/access.log")
+             (output "/var/log/anonip/access.log")
+             (debug? #t)))))
+
+(define (run-anonip-test)
+  (define os
+    (marionette-operating-system
+     %anonip-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     ;; We are interested in verifying if anonip still launches following a
+     ;; reboot; thus make the base image writable.
+     (volatile? #f)))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (srfi srfi-64)
+                       (gnu build marionette))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "anonip")
+
+          (test-assert "service is running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (wait-for-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-assert "service can be restarted"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (restart-service 'anonip-/var/log/anonip/access.log)
+                (wait-for-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-assert "ip addresses are anonymized"
+            (marionette-eval
+             '(begin
+                (use-modules (ice-9 textual-ports))
+                (call-with-output-file "/var/run/anonip/access.log"
+                  (lambda (port)
+                    (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+                    (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+                (#$retry-on-error
+                 (lambda ()
+                   (call-with-input-file "/var/log/anonip/access.log"
+                     (lambda (port)
+                       (let ((content (get-string-all port)))
+                         ;; The expected values are taken from anonip's test
+                         ;; suite (see its test_module.py file).
+                         (or (and (string-contains content "192.168.96.0")
+                                  (string-contains content "2001:db8:85a0::"))
+                             (error "could not find expected anonymized IPs"
+                                    content))))))
+                 #:times 20
+                 #:delay 1))
+             marionette))
+
+          (test-assert "service is running after reboot"
+            (begin
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (eval-there '(begin
+                                 (use-modules (shepherd system))
+                                 (sync) ;ensure the log is fully written
+                                 (reboot))))
+               marionette)
+              ;; Note: a distinct marionette-eval call is needed here; if
+              ;; included in the previous one issuing the reboot,
+              ;; 'wait-for-service' would apparently run before the system had
+              ;; rebooted (and succeed), which would defeat the test.
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (wait-for-service 'anonip-/var/log/anonip/access.log))
+               marionette)))
+
+          (test-assert "service can be stopped"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (stop-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-end))))
+
+  (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+  (system-test
+   (name "anonip")
+   (description "Anonymize logs via Anonip")
+   (value (run-anonip-test))))
+
 \f
 ;;;
 ;;; Patchwork
-- 
2.46.0

[bug#74151] [PATCH v2 8/9] tests: web: Add nginx+anonip test.

[Maxim Cournoyer]

This test reproduces the problem reported in bug #59181 ("[berlin] web
services fail to start on reboot due to anonip").

* gnu/tests/web.scm (run-webserver-test) <extra-args>: New keyword argument.
(%nginx+anonip-os, nginx-anonip-tests, %test-nginx+anonip): New variables.

Change-Id: I7bbc8fb9f2ab33ce89bf1c0945d7ddbddf26a7ea
---
New commit in v2

 gnu/tests/web.scm | 43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 6ef32138ed..33e0a697a8 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -50,6 +50,7 @@ (define-module (gnu tests web)
   #:use-module (ice-9 match)
   #:export (%test-httpd
             %test-nginx
+            %test-nginx+anonip
             %test-varnish
             %test-php-fpm
             %test-hpcguix-web
@@ -91,9 +92,11 @@ (define retry-on-error
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
 
-(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
-  "Run tests in %NGINX-OS, which has nginx running and listening on
-HTTP-PORT."
+(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080)
+                             extra-tests)
+  "Run tests in %NGINX-OS, which has nginx running and listening on HTTP-PORT.
+EXTRA-TESTS should be a sexp of gexp containing extra code to run as part of
+the tests."
   (define os
     (marionette-operating-system
      test-os
@@ -154,6 +157,8 @@ (define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
                       marionette)))
                  '())
 
+          #$extra-tests
+
           (test-end))))
 
   (gexp->derivation (string-append name "-test") test))
@@ -209,6 +214,38 @@ (define %test-nginx
    (value (run-webserver-test name %nginx-os
                               #:log-file "/var/log/nginx/access.log"))))
 
+(define %nginx+anonip-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/access.log")
+             (output "/var/log/anonip/access.log")
+             (debug? #t)))
+   (service nginx-service-type
+            (nginx-configuration
+             (log-directory "/var/run/anonip/")
+             (server-blocks %nginx-servers)
+             (shepherd-requirement '(anonip-/var/log/anonip/access.log))))
+   (simple-service 'make-http-root activation-service-type
+                   %make-http-root)))
+
+(define nginx-anonip-tests
+  #~(test-assert "anonip service is running"
+      (marionette-eval
+       '(begin
+          (use-modules (gnu services herd))
+          (wait-for-service 'anonip-/var/log/anonip/access.log))
+       marionette)))
+
+(define %test-nginx+anonip
+  (system-test
+   (name "nginx+anonip")
+   (description "Run a NGINX server with logs anonymized by Anonip")
+   (value (run-webserver-test "nginx" %nginx+anonip-os
+                              #:log-file "/var/log/anonip/access.log"
+                              #:extra-tests nginx-anonip-tests))))
+
 \f
 ;;;
 ;;; Varnish
-- 
2.46.0

[bug#74151] [PATCH v2 9/9] services: web: Fix race between nginx activation and anonip.

[Maxim Cournoyer]

* gnu/services/web.scm (anonip-shepherd-service): Recreate the input file when
it's not a FIFO.

Fixes: <https://issues.guix.gnu.org/59181>
Change-Id: I8ba87f9fc48ecfd515e34bdee9e2949a2a559f9c
---
New commit in v2

 gnu/services/web.scm | 64 ++++++++++++++++++++------------------------
 1 file changed, 29 insertions(+), 35 deletions(-)

diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cf3515bf70..4cf7c68997 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -1499,41 +1499,35 @@ (define (anonip-shepherd-service config)
        "Anonimyze the given log file location with anonip.")
       (start
        #~(lambda ()
-           (define (spawn)
-             (fork+exec-command
-              (append
-               (list #$(file-append (anonip-configuration-anonip config)
-                                    "/bin/anonip")
-                     (string-append "--input=" #$input)
-                     (string-append "--output=" #$output))
-               (if #$(anonip-configuration-debug? config)
-                   '("--debug") (list))
-               (if #$(anonip-configuration-skip-private? config)
-                   '("--skip-private") (list))
-               '#$(optional anonip-configuration-column "--column")
-               '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
-               '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
-               '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement "--replacement")
-               '#$(optional anonip-configuration-delimiter "--delimiter")
-               '#$(optional anonip-configuration-regex "--regex"))
-              ;; Run in a UTF-8 locale
-              #:environment-variables
-              (list (string-append "GUIX_LOCPATH="
-                                   #$(libc-utf8-locales-for-target)
-                                   "/lib/locale")
-                    "LC_ALL=en_US.utf8")))
-
-           (let ((stat (stat #$input #f)))
-             (cond ((not stat)
-                    (mknod #$input 'fifo #o600 0)
-                    (spawn))
-                   ((eq? 'fifo (stat:type stat))
-                    (spawn))
-                   (else
-                    (format #t "'~a' is not a FIFO; bailing out~%"
-                            #$input)
-                    #f)))))
+           ;; Always attempt to recreate the named pipe, as activation scripts
+           ;; such as that of nginx may have created plain files in its place
+           ;; (see: https://issues.guix.gnu.org/59181).
+           (false-if-exception (delete-file #$input))
+           (mknod #$input 'fifo #o600 0)
+
+           (fork+exec-command
+            (append
+             (list #$(file-append (anonip-configuration-anonip config)
+                                  "/bin/anonip")
+                   (string-append "--input=" #$input)
+                   (string-append "--output=" #$output))
+             (if #$(anonip-configuration-debug? config)
+                 '("--debug") (list))
+             (if #$(anonip-configuration-skip-private? config)
+                 '("--skip-private") (list))
+             '#$(optional anonip-configuration-column "--column")
+             '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
+             '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
+             '#$(optional anonip-configuration-increment "--increment")
+             '#$(optional anonip-configuration-replacement "--replacement")
+             '#$(optional anonip-configuration-delimiter "--delimiter")
+             '#$(optional anonip-configuration-regex "--regex"))
+            ;; Run in a UTF-8 locale
+            #:environment-variables
+            (list (string-append "GUIX_LOCPATH="
+                                 #$(libc-utf8-locales-for-target)
+                                 "/lib/locale")
+                  "LC_ALL=en_US.utf8"))))
       (stop #~(make-kill-destructor))))))
 
 (define anonip-service-type
-- 
2.46.0

[bug#74151] [PATCH 0/7] Add anonip system test.

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> This was developed as part of investigating the source of bug #59181 ([berlin]
> web services fail to start on reboot due to anonip).
>
> There is no issue having anonip run and the system reboot (as should be
> expected as /var/run is supposed to be cleared thus the named pipes should be
> recreated every boot).  This points to a race with nginx that would "win"
> creating its log files under /var/run/anonip/* before the anonip service does.
>
> It'll need further investigating in that direction.

This is nice work, LGTM!

This did not elucidate the cause of #59181, right?

Thanks,
Ludo’.