From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.0.0.0/16 X-Spam-Status: No, score=-3.2 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out0.migadu.com (out0.migadu.com [IPv6:2001:41d0:2:267::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 8AB411F934 for ; Sat, 10 Apr 2021 19:49:53 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kyleam.com; s=key1; t=1618084188; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=GFlsniVbazIpu8vK+bZauc6BPcv3Z+UM3mXTt95jAhk=; b=kKpZ1LU3feq5ivkJTREcpQmbbXaXD1YBBngi0BF+086mR1mRpzC9WOjSZZf3gYD4dq+ktk jcaSejK3gkjoSuI2Vwu9z25fNAgLlPJL5Lu91u+53csBYh01ISbpKITz63e/eNOn1hxWl5 XWYz9DzXcUdbFG0WnpRXeiQtlFNmlWlDjKSu/1o3IYgYg9IufKInaBmq393B7yc7BVqY25 bRW9Q4gcD72E1hlKLgxzCQiuYeXsY6KJRq2pDz00kXfRiDC/PFatxV5k7b6hVqALBHZygI vPAOYL6HACLPAqZT+09UW66e4D7NjNUpWSLpBjG2NL0p7UWRqq+3YOFPzMBZ1g== From: Kyle Meyer To: Eric Wong Cc: meta@public-inbox.org Subject: Re: archive links broken with obfuscate=true In-Reply-To: <20210410051550.GA4654@dcvr> References: <87a6q8p5qa.fsf@kyleam.com> <20210409102129.GA16787@dcvr> <87zgy7rs9q.fsf@kyleam.com> <20210409233700.GA11190@dcvr> <87sg3ysrzu.fsf@kyleam.com> <20210410051550.GA4654@dcvr> Date: Sat, 10 Apr 2021 15:49:46 -0400 Message-ID: <87wnt9or6t.fsf@kyleam.com> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: kyle@kyleam.com List-Id: Eric Wong writes: > Kyle Meyer wrote: >> Eric Wong writes: >> >> > Have you run any performance tests? >> >> No. To get an idea of how to approach that, would you suggest I look at >> xt/perf-msgview.t? > > Yeah, probably that with some tweaks; or running -httpd with ab, > wrk or some other HTTP benchmark that uses persistent connections. > > I'm OK with things being slower with this option enabled, but > not with trivial denial-of-service vectors. Below is my initial attempt at tweaking xt/perf-msgview.t. xt/perf-obfuscate.t enables obfuscation in the inbox if PI_OBFUSCATE is set in the environment: $ PI_OBFUSCATE=1 GIANT_INBOX_DIR=/tmp/test perl xt/perf-obfuscate.t Here are my unscientific timings for three cases: 1) no obfuscation, 2) obfuscation on the current master (ea4e9025dd), and 3) obfuscation with the patch upthread. The inbox is set to the Org mode's list archives (), which at the time of execution contained 135,885 messages. | obfuscate | run | wall | usr | sys | |-------------+-----+------+--------+------| | no | 1 | 50 | 49.14 | 0.57 | | no | 2 | 49 | 47.76 | 0.58 | | | | | | | | yes, master | 1 | 56 | 54.47 | 0.58 | | yes, master | 2 | 55 | 54.24 | 0.58 | | | | | | | | yes, patch | 1 | 175 | 174.71 | 0.52 | | yes, patch | 2 | 176 | 174.33 | 0.56 | diff --git a/xt/perf-obfuscate.t b/xt/perf-obfuscate.t new file mode 100644 index 00000000..2a8d5c1e --- /dev/null +++ b/xt/perf-obfuscate.t @@ -0,0 +1,64 @@ +# Copyright (C) 2019-2021 all contributors +# License: AGPL-3.0+ +use strict; +use warnings; +use Test::More; +use Benchmark qw(:all); +use PublicInbox::Inbox; +use PublicInbox::View; +use PublicInbox::TestCommon; + +my $inboxdir = $ENV{GIANT_INBOX_DIR}; +plan skip_all => "GIANT_INBOX_DIR not defined for $0" unless $inboxdir; + +my $obfuscate = $ENV{PI_OBFUSCATE} ? 1 : 0; +print "obfuscate=$obfuscate\n"; + +my @cat = qw(cat-file --buffer --batch-check --batch-all-objects); +if (require_git(2.19, 1)) { + push @cat, '--unordered'; +} else { + warn +"git <2.19, cat-file lacks --unordered, locality suffers\n"; +} +require_mods qw(Plack::Util); +use_ok 'Plack::Util'; +my $ibx = PublicInbox::Inbox->new({ inboxdir => $inboxdir, name => 'name' , + obfuscate => $obfuscate}); +my $git = $ibx->git; +my $fh = $git->popen(@cat); +my $vec = ''; +vec($vec, fileno($fh), 1) = 1; +select($vec, undef, undef, 60) or die "timed out waiting for --batch-check"; + +my $ctx = { + env => { HTTP_HOST => 'example.com', 'psgi.url_scheme' => 'https' }, + ibx => $ibx, + www => Plack::Util::inline_object(style => sub {''}), +}; +my ($mime, $res, $oid, $type); +my $n = 0; +my $obuf = ''; +my $m = 0; + +my $cb = sub { + $mime = PublicInbox::Eml->new(shift); + PublicInbox::View::multipart_text_as_html($mime, $ctx); + ++$m; + $obuf = ''; +}; + +my $t = timeit(1, sub { + $ctx->{obuf} = \$obuf; + $ctx->{mhref} = '../'; + while (<$fh>) { + ($oid, $type) = split / /; + next if $type ne 'blob'; + ++$n; + $git->cat_async($oid, $cb); + } + $git->cat_async_wait; +}); +diag 'multipart_text_as_html took '.timestr($t)." for $n <=> $m messages"; +is($m, $n, 'rendered all messages'); +done_testing();