[bug#32771] [PATCH 0/2] Varnish service

[bug#32771] [PATCH 0/2] Varnish service

[Marius Bakke]

These patches adds a service for the Varnish HTTP proxy.

Marius Bakke (2):
  gnu: varnish: Use absolute file name of "rm".
  services: Add Varnish service.

 doc/guix.texi        |  82 +++++++++++++++++++++++++++++++++
 gnu/packages/web.scm |   4 +-
 gnu/services/web.scm | 107 ++++++++++++++++++++++++++++++++++++++++++-
 gnu/tests/web.scm    |  41 +++++++++++++++++
 4 files changed, 232 insertions(+), 2 deletions(-)

-- 
2.19.0

[bug#32771] [PATCH 1/2] gnu: varnish: Use absolute file name of "rm".

[Marius Bakke]

* gnu/packages/web.scm (varnish)[arguments]: Rename 'patch-bin-sh-phase to
'use-absolute-file-names and add substitution.
---
 gnu/packages/web.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index b6bee57f9..26c2e9128 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -5019,12 +5019,14 @@ deployments.")
                                "--localstatedir=/var")
        #:phases
        (modify-phases %standard-phases
-         (add-after 'unpack 'patch-/bin/sh
+         (add-after 'unpack 'use-absolute-file-names
            (lambda _
              (substitute* '("bin/varnishtest/vtc_varnish.c"
                             "bin/varnishtest/vtc_process.c"
                             "bin/varnishd/mgt/mgt_vcc.c")
                (("/bin/sh") (which "sh")))
+             (substitute* "bin/varnishd/mgt/mgt_shmem.c"
+               (("rm -rf") (string-append (which "rm") " -rf")))
              #t))
          (add-before 'install 'patch-Makefile
            (lambda _
-- 
2.19.0

[bug#32771] [PATCH 2/2] services: Add Varnish service.

[Marius Bakke]

* gnu/services/web.scm (<varnish-configuration>): New record type.
(%varnish-accounts, %varnish-activation, varnish-service-type): New variables.
(varnish-shepherd-service): New procedure.
* gnu/tests/web.scm (%varnish-vcl, %varnish-os): New variables.
(%test-varnish): New test.
* doc/guix.texi (Web Services): Document it.
---
 doc/guix.texi        |  82 +++++++++++++++++++++++++++++++++
 gnu/services/web.scm | 107 ++++++++++++++++++++++++++++++++++++++++++-
 gnu/tests/web.scm    |  41 +++++++++++++++++
 3 files changed, 229 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 8987b20fa..543b7d4f7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -16709,6 +16709,88 @@ body of a named location block cannot contain location blocks.
 @end table
 @end deftp
 
+@subsubheading Varnish Cache
+@cindex Varnish
+Varnish is a fast cache server that sits in between web applications
+and end users.  It proxies requests from clients and caches the
+accessed URLs such that multiple requests for the same resource only
+creates one request to the back-end.
+
+@defvr {Scheme Variable} varnish-service-type
+A service type for the Varnish daemon.
+@end defvr
+
+@deftp {Data Type} varnish-configuration
+Data type representing the @code{varnish} service configuration.
+This type has the following parameters:
+
+@table @asis
+@item @code{package} (default: @code{varnish})
+The Varnish package to use.
+
+@item @code{name} (default: @code{"default"})
+A name for this Varnish instance.  Varnish will create a directory in
+@file{/var/varnish/} with this name and keep temporary files there.  If
+the name starts with a forward slash, it is interpreted as an absolute
+directory name.
+
+Pass the @code{-n} argument to other Varnish programs to connect to the
+named instance, e.g. @command{varnishncsa -n default}.
+
+@item @code{backend} (default: @code{"localhost:8080"})
+The backend to use.  This option has no effect if @code{vcl} is set.
+
+@item @code{vcl} (default: #f)
+The @dfn{VCL} (Varnish Configuration Language) program to run.  If this
+is @code{#f}, Varnish will proxy @code{backend} using the default
+configuration.  Otherwise this must be a file-like object with valid
+VCL syntax.
+
+@c Varnish does not support HTTPS, so keep this URL to avoid confusion.
+For example, to mirror @url{http://www.gnu.org,www.gnu.org} with VCL you
+can do something along these lines:
+
+@example
+(define %gnu-mirror
+  (plain-file
+   "gnu.vcl"
+   "vcl 4.1;
+backend gnu @{ .host = "www.gnu.org"; @}"))
+
+(operating-system
+  ...
+  (services (cons (service varnish-service-type
+                           (varnish-configuration
+                            (listen '(":80"))
+                            (vcl %gnu-mirror)))
+                  %base-services)))
+@end example
+
+The configuration of an already running Varnish instance can be inspected
+and changed using the @command{varnishadm} program.
+
+Consult the @url{https://varnish-cache.org/docs/,Varnish User Guide} and
+@url{https://book.varnish-software.com/4.0/,Varnish Book} for
+comprehensive documentation on Varnish and its configuration language.
+
+@item @code{listen} (default: @code{'("localhost:80")})
+List of addresses Varnish will listen on.
+
+@item @code{storage} (default: @code{'()})
+List of storage backends that will be available in VCL.  The first backend
+becomes the default.  If left empty, Varnish will choose
+@code{'("malloc,unlimited")}.
+
+@item @code{parameters} (default: @code{'()})
+List of run-time parameters in the form @code{'(("parameter" . "value"))}.
+
+@item @code{extra-options} (default: @code{'()})
+Additional arguments to pass to the @command{varnishd} process.
+
+@end table
+@end deftp
+
+@subsubheading FastCGI
 @cindex fastcgi
 @cindex fcgiwrap
 FastCGI is an interface between the front-end and the back-end of a web
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index 1c38e7d8d..52358acce 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2017, 2018 Clément Lassieur <clement@lassieur.org>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
 ;;; Copyright © 2017 Christopher Baines <mail@cbaines.net>
+;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -190,7 +191,21 @@
             tailon-configuration-config-file
             tailon-configuration-package
 
-            tailon-service-type))
+            tailon-service-type
+
+            <varnish-configuration>
+            varnish-configuration
+            varnish-configuration?
+            varnish-configuration-package
+            varnish-configuration-name
+            varnish-configuration-backend
+            varnish-configuration-vcl
+            varnish-configuration-listen
+            varnish-configuration-storage
+            varnish-configuration-parameters
+            varnish-configuration-extra-options
+
+            varnish-service-type))
 
 ;;; Commentary:
 ;;;
@@ -1162,3 +1177,93 @@ files.")
                   (files (append (tailon-configuration-file-files old-config-file)
                                  files))))))))
    (default-value (tailon-configuration))))
+
+\f
+;;;
+;;; Varnish
+;;;
+
+(define-record-type* <varnish-configuration>
+  varnish-configuration make-varnish-configuration
+  varnish-configuration?
+  (package             varnish-configuration-package          ;<package>
+                       (default varnish))
+  (name                varnish-configuration-name             ;string
+                       (default "default"))
+  (backend             varnish-configuration-backend          ;string
+                       (default "localhost:8080"))
+  (vcl                 varnish-configuration-vcl              ;#f | <file-like>
+                       (default #f))
+  (listen              varnish-configuration-listen           ;list of strings
+                       (default '("localhost:80")))
+  (storage             varnish-configuration-storage          ;list of strings
+                       (default '()))
+  (parameters          varnish-configuration-parameters       ;list of pairs
+                       (default '()))
+  (extra-options       varnish-configuration-extra-options    ;list of strings
+                       (default '())))
+
+(define %varnish-accounts
+  (list (user-group
+         (name "varnish")
+         (system? #t))
+        (user-account
+         (name "varnish")
+         (group "varnish")
+         (system? #t)
+         (comment "Varnish Cache User")
+         (home-directory "/var/varnish")
+         (shell (file-append shadow "/sbin/nologin")))))
+
+(define %varnish-activation
+  #~(begin
+      (use-modules (guix build utils))
+      (let ((home-dir "/var/varnish")
+            (user (getpwnam "varnish")))
+        (mkdir-p home-dir)
+        (chown home-dir (passwd:uid user) (passwd:gid user))
+        (chmod home-dir #o755))))
+
+(define varnish-shepherd-service
+  (match-lambda
+    (($ <varnish-configuration> package name backend vcl listen storage
+                                parameters extra-options)
+     (list (shepherd-service
+            (provision (list (symbol-append 'varnish- (string->symbol name))))
+            (documentation (string-append "The Varnish Web Accelerator"
+                                          " (" name ")"))
+            (requirement '(networking))
+            (start #~(make-forkexec-constructor
+                      (list #$(file-append package "/sbin/varnishd")
+                            "-n" #$name
+                            #$@(if vcl
+                                   #~("-f" #$vcl)
+                                   #~("-b" #$backend))
+                            #$@(append-map (lambda (a) (list "-a" a)) listen)
+                            #$@(append-map (lambda (s) (list "-s" s)) storage)
+                            #$@(append-map (lambda (p)
+                                             (list "-p" (format #f "~a=~a"
+                                                                (car p) (cdr p))))
+                                           parameters)
+                            #$@extra-options)
+                      ;; Varnish will drop privileges to the "varnish" user when
+                      ;; it exists.  Not passing #:user here allows the service
+                      ;; to bind to ports < 1024.
+                      #:pid-file (if (string-prefix? "/" #$name)
+                                     (string-append #$name "/_.pid")
+                                     (string-append "/var/varnish/" #$name "/_.pid"))))
+            (stop #~(make-kill-destructor)))))))
+
+(define varnish-service-type
+  (service-type
+   (name 'varnish)
+   (description "Run the Varnish cache server.")
+   (extensions
+    (list (service-extension account-service-type
+                             (const %varnish-accounts))
+          (service-extension activation-service-type
+                             (const %varnish-activation))
+          (service-extension shepherd-root-service-type
+                             varnish-shepherd-service)))
+   (default-value
+     (varnish-configuration))))
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 45fcb668f..bcc919137 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -32,6 +32,7 @@
   #:use-module (guix store)
   #:export (%test-httpd
             %test-nginx
+            %test-varnish
             %test-php-fpm
             %test-hpcguix-web
             %test-tailon))
@@ -168,6 +169,46 @@ HTTP-PORT."
                               #:log-file "/var/log/nginx/access.log"))))
 
 \f
+;;;
+;;; Varnish
+;;;
+
+(define %varnish-vcl
+  (mixed-text-file
+   "varnish-test.vcl"
+   "vcl 4.0;
+backend dummy { .host = \"127.1.1.1\"; }
+sub vcl_recv { return(synth(200, \"OK\")); }
+sub vcl_synth {
+  synthetic(\"" %index.html-contents "\");
+  set resp.http.Content-Type = \"text/plain\";
+  return(deliver);
+}"))
+
+(define %varnish-os
+  (simple-operating-system
+   (dhcp-client-service)
+   ;; Pretend to be a web server that serves %index.html-contents.
+   (service varnish-service-type
+            (varnish-configuration
+             (name "/tmp/server")
+             ;; Use a small VSL buffer to fit in the test VM.
+             (parameters '(("vsl_space" . "4M")))
+             (vcl %varnish-vcl)))
+   ;; Proxy the "server" using the builtin configuration.
+   (service varnish-service-type
+            (varnish-configuration
+             (parameters '(("vsl_space" . "4M")))
+             (backend "localhost:80")
+             (listen '(":8080"))))))
+
+(define %test-varnish
+  (system-test
+   (name "varnish")
+   (description "Test the Varnish Cache server.")
+   (value (run-webserver-test "varnish-default" %varnish-os))))
+
+\f
 ;;;
 ;;; PHP-FPM
 ;;;
-- 
2.19.0

[bug#32771] [PATCH 2/2] services: Add Varnish service.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2506 bytes --]

Hey Marius,

I've not got much experience with Varnish, but this patch applies, and
the system test passes :)

Marius Bakke <mbakke@fastmail.com> writes:

> +@subsubheading Varnish Cache
> +@cindex Varnish
> +Varnish is a fast cache server that sits in between web applications
> +and end users.  It proxies requests from clients and caches the
> +accessed URLs such that multiple requests for the same resource only
> +creates one request to the back-end.
> +
> +@defvr {Scheme Variable} varnish-service-type
> +A service type for the Varnish daemon.
> +@end defvr

Given there are not other service types for Varnish in Guix, "The
service type ..." would probably be clearer here, or just "Service type
...".

> +@deftp {Data Type} varnish-configuration
> +Data type representing the @code{varnish} service configuration.
> +This type has the following parameters:
> +
> +@table @asis
> +@item @code{package} (default: @code{varnish})
> +The Varnish package to use.
> +
> +@item @code{name} (default: @code{"default"})
> +A name for this Varnish instance.  Varnish will create a directory in
> +@file{/var/varnish/} with this name and keep temporary files there.  If
> +the name starts with a forward slash, it is interpreted as an absolute
> +directory name.

Most services in Guix use /var/lib for data, would this work for
Varnish?

Also, I wonder if you'd considered supporing running multiple instances
of varnishd, I guess the "name" might come in useful then.

> +@c Varnish does not support HTTPS, so keep this URL to avoid confusion.
> +For example, to mirror @url{http://www.gnu.org,www.gnu.org} with VCL you
> +can do something along these lines:

Does "@c" mean a comment?

> +(define %varnish-accounts
> +  (list (user-group
> +         (name "varnish")
> +         (system? #t))
> +        (user-account
> +         (name "varnish")
> +         (group "varnish")
> +         (system? #t)
> +         (comment "Varnish Cache User")
> +         (home-directory "/var/varnish")
> +         (shell (file-append shadow "/sbin/nologin")))))
> +
> +(define %varnish-activation
> +  #~(begin
> +      (use-modules (guix build utils))
> +      (let ((home-dir "/var/varnish")
> +            (user (getpwnam "varnish")))
> +        (mkdir-p home-dir)
> +        (chown home-dir (passwd:uid user) (passwd:gid user))
> +        (chmod home-dir #o755))))

Is this necessary, as I think the users home directory might be
automatically created?

Anyway, this looks pretty much good to me.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#32771: [PATCH 2/2] services: Add Varnish service.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3159 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> Hey Marius,
>
> I've not got much experience with Varnish, but this patch applies, and
> the system test passes :)
>
> Marius Bakke <mbakke@fastmail.com> writes:
>
>> +@subsubheading Varnish Cache
>> +@cindex Varnish
>> +Varnish is a fast cache server that sits in between web applications
>> +and end users.  It proxies requests from clients and caches the
>> +accessed URLs such that multiple requests for the same resource only
>> +creates one request to the back-end.
>> +
>> +@defvr {Scheme Variable} varnish-service-type
>> +A service type for the Varnish daemon.
>> +@end defvr
>
> Given there are not other service types for Varnish in Guix, "The
> service type ..." would probably be clearer here, or just "Service type
> ...".

Just "Service type ..." is better, thanks!

>> +@deftp {Data Type} varnish-configuration
>> +Data type representing the @code{varnish} service configuration.
>> +This type has the following parameters:
>> +
>> +@table @asis
>> +@item @code{package} (default: @code{varnish})
>> +The Varnish package to use.
>> +
>> +@item @code{name} (default: @code{"default"})
>> +A name for this Varnish instance.  Varnish will create a directory in
>> +@file{/var/varnish/} with this name and keep temporary files there.  If
>> +the name starts with a forward slash, it is interpreted as an absolute
>> +directory name.
>
> Most services in Guix use /var/lib for data, would this work for
> Varnish?

Probably, although I didn't bother trying it.  Those files are temporary
anyway, perhaps /var/cache/varnish would be better?  But, I think the
upstream default is OK.

> Also, I wonder if you'd considered supporing running multiple instances
> of varnishd, I guess the "name" might come in useful then.

If you read the system test closely, you'll notice it does actually run
multiple instances, one proxying the other, and testing different
aspects of the service definition :-)

>> +@c Varnish does not support HTTPS, so keep this URL to avoid confusion.
>> +For example, to mirror @url{http://www.gnu.org,www.gnu.org} with VCL you
>> +can do something along these lines:
>
> Does "@c" mean a comment?

Yes.

>> +(define %varnish-accounts
>> +  (list (user-group
>> +         (name "varnish")
>> +         (system? #t))
>> +        (user-account
>> +         (name "varnish")
>> +         (group "varnish")
>> +         (system? #t)
>> +         (comment "Varnish Cache User")
>> +         (home-directory "/var/varnish")
>> +         (shell (file-append shadow "/sbin/nologin")))))
>> +
>> +(define %varnish-activation
>> +  #~(begin
>> +      (use-modules (guix build utils))
>> +      (let ((home-dir "/var/varnish")
>> +            (user (getpwnam "varnish")))
>> +        (mkdir-p home-dir)
>> +        (chown home-dir (passwd:uid user) (passwd:gid user))
>> +        (chmod home-dir #o755))))
>
> Is this necessary, as I think the users home directory might be
> automatically created?

You are correct!  I removed the activation script.

> Anyway, this looks pretty much good to me.

Thank you very much for reviewing :-)

Pushed as 3b97a1779f3b65d582b8edc8c154b6414314b946.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#32771] [PATCH 2/2] services: Add Varnish service.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2007 bytes --]


Marius Bakke <mbakke@fastmail.com> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Also, I wonder if you'd considered supporing running multiple instances
>> of varnishd, I guess the "name" might come in useful then.
>
> If you read the system test closely, you'll notice it does actually run
> multiple instances, one proxying the other, and testing different
> aspects of the service definition :-)

Huh, I made this comment based off the service itself. I didn't quite
realise you could just have multiple services of the same type without
handling that explicitly in the service type.

>>> +@c Varnish does not support HTTPS, so keep this URL to avoid confusion.
>>> +For example, to mirror @url{http://www.gnu.org,www.gnu.org} with VCL you
>>> +can do something along these lines:
>>
>> Does "@c" mean a comment?
>
> Yes.

Ah, I think I get it now. I was reading this comment as being the whole
block, but now I see it's only the line about the use of HTTP.

>>> +(define %varnish-accounts
>>> +  (list (user-group
>>> +         (name "varnish")
>>> +         (system? #t))
>>> +        (user-account
>>> +         (name "varnish")
>>> +         (group "varnish")
>>> +         (system? #t)
>>> +         (comment "Varnish Cache User")
>>> +         (home-directory "/var/varnish")
>>> +         (shell (file-append shadow "/sbin/nologin")))))
>>> +
>>> +(define %varnish-activation
>>> +  #~(begin
>>> +      (use-modules (guix build utils))
>>> +      (let ((home-dir "/var/varnish")
>>> +            (user (getpwnam "varnish")))
>>> +        (mkdir-p home-dir)
>>> +        (chown home-dir (passwd:uid user) (passwd:gid user))
>>> +        (chmod home-dir #o755))))
>>
>> Is this necessary, as I think the users home directory might be
>> automatically created?
>
> You are correct!  I removed the activation script.

Great :)

>> Anyway, this looks pretty much good to me.
>
> Thank you very much for reviewing :-)
>
> Pushed as 3b97a1779f3b65d582b8edc8c154b6414314b946.

Awesome :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

[bug#32771] [PATCH 1/2] gnu: varnish: Use absolute file name of "rm".

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]


Marius Bakke <mbakke@fastmail.com> writes:

> * gnu/packages/web.scm (varnish)[arguments]: Rename 'patch-bin-sh-phase to
> 'use-absolute-file-names and add substitution.
> ---
>  gnu/packages/web.scm | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
> index b6bee57f9..26c2e9128 100644
> --- a/gnu/packages/web.scm
> +++ b/gnu/packages/web.scm
> @@ -5019,12 +5019,14 @@ deployments.")
>                                 "--localstatedir=/var")
>         #:phases
>         (modify-phases %standard-phases
> -         (add-after 'unpack 'patch-/bin/sh
> +         (add-after 'unpack 'use-absolute-file-names
>             (lambda _
>               (substitute* '("bin/varnishtest/vtc_varnish.c"
>                              "bin/varnishtest/vtc_process.c"
>                              "bin/varnishd/mgt/mgt_vcc.c")
>                 (("/bin/sh") (which "sh")))
> +             (substitute* "bin/varnishd/mgt/mgt_shmem.c"
> +               (("rm -rf") (string-append (which "rm") " -rf")))
>               #t))
>           (add-before 'install 'patch-Makefile
>             (lambda _

This applies and varnish builds, so this looks good to me :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]