[PATCH 0/2] performance-related notes+docs

[PATCH 0/2] performance-related notes+docs

[Eric Wong]

Opposing goals here, yes, I'm hell bent on sticking to a
scripting language for hackability and eliminating the need to
provide reproducible builds.  On the other hand, this should
be able to scale to handle LKML.

Eric Wong (2):
      TODO: update documentation for performance items
      examples: add varnish-4.vcl

 TODO                   | 11 +++++++-
 examples/varnish-4.vcl | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+), 1 deletion(-)

[PATCH 1/2] TODO: update documentation for performance items

[Eric Wong]

---
 TODO | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/TODO b/TODO
index 3b6401f..30ba38b 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,11 @@
 TODO items for public-inbox
 
-(Not in any particular order)
+(Not in any particular order, and
+performance, ease-of-setup, installation, maintainability, etc
+all need to be considered for everything we introduce)
+
+* general performance improvements, but without relying on
+  XS or compiled code any more than we currently do.
 
 * mailmap support (same as git) for remapping expired email addresses
 
@@ -45,6 +50,10 @@ TODO items for public-inbox
 * Consider storing git blob ID in Xapian doc data to avoid ref
   and tree lookups based on Message-Id.
 
+* streaming Email::MIME replacement: currently we generate many
+  allocations/strings for headers we never look at and slurp
+  entire message bodies into memory.
+
 * Allow in-place Xapian updates without clobbering the whole
   index (versioning each doc data entry?) for big archives
 
-- 
EW

[PATCH 2/2] examples: add varnish-4.vcl

[Eric Wong]

Well, I'm fumbling along with this config.  Might as well
fumble along with it publically :)
---
 examples/varnish-4.vcl | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 examples/varnish-4.vcl

diff --git a/examples/varnish-4.vcl b/examples/varnish-4.vcl
new file mode 100644
index 0000000..7439679
--- /dev/null
+++ b/examples/varnish-4.vcl
@@ -0,0 +1,74 @@
+# Example VCL for Varnish 4.0 with public-inbox WWW code
+# This is based on what shipped for 3.x a long time ago (I think)
+# and I'm hardly an expert in VCL (nor should we expect anybody
+# who maintains a public-inbox HTTP interface to be).
+#
+# It seems to work for providing some protection from traffic
+# bursts; but perhaps the public-inbox WWW interface can someday
+# provide enough out-of-the-box performance that configuration
+# of an extra component is pointless.
+
+vcl 4.0;
+backend default {
+	.host = "127.0.0.1";
+	.port = "280";
+}
+
+sub vcl_recv {
+	if (req.restarts == 0) {
+		if (req.http.x-forwarded-for) {
+			set req.http.X-Forwarded-For =
+			req.http.X-Forwarded-For + ", " + client.ip;
+		} else {
+			set req.http.X-Forwarded-For = client.ip;
+		}
+	}
+	if (req.method != "GET" &&
+			req.method != "HEAD" &&
+			req.method != "PUT" &&
+			req.method != "POST" &&
+			req.method != "TRACE" &&
+			req.method != "OPTIONS" &&
+			req.method != "DELETE") {
+		/* Non-RFC2616 or CONNECT which is weird. */
+		return (pipe);
+	}
+	if (req.method != "GET" && req.method != "HEAD") {
+		/* We only deal with GET and HEAD by default */
+		return (pass);
+	}
+	if (req.http.Authorization || req.http.Cookie) {
+		/* Not cacheable by default */
+		return (pass);
+	}
+	return (hash);
+}
+
+sub vcl_hash {
+	hash_data(req.url);
+	if (req.http.host) {
+		hash_data(req.http.host);
+	} else {
+		hash_data(server.ip);
+	}
+	if (req.http.X-Forwarded-Proto) {
+		hash_data(req.http.X-Forwarded-Proto);
+	}
+	return (lookup);
+}
+
+sub vcl_backend_response {
+	set beresp.grace = 60s;
+	set beresp.do_stream = true;
+	if (beresp.ttl <= 0s ||
+		beresp.http.Set-Cookie ||
+		beresp.http.Vary == "*") {
+		/* Mark as "Hit-For-Pass" for the next 2 minutes */
+		set beresp.ttl = 120 s;
+		set beresp.uncacheable = true;
+		return (deliver);
+	} else {
+		set beresp.ttl = 10s;
+	}
+	return (deliver);
+}
-- 
EW