[PATCH 0/2] begin online help for each public-inbox

[PATCH 0/2] begin online help for each public-inbox

[Eric Wong]

I expect this will be tweaked and edited heavily in coming
weeks, but this at least a start.

I'll be keeping everything within the constraints of a
plain-text email, so no cheating with styling or #fragment
tags, and definitely no deceptive hiding of URLs to external
resources.

Fortunately, deploying changes are cheap and relatively
risk-free with zero user-visible downtime
(which also needs to be documented :x)

Thanks to Philip Oakley at the git list for reminding me the
design_www document is actually helpful to users, not just
hackers :)

  https://public-inbox.org/git/D5A2E231FFE74D1C891A1653E1C2D797@PhilipOakley/

Eric Wong (2):
  www: implement generic help text
  searchview: link to internal help text

 MANIFEST                      |   2 +
 lib/PublicInbox/Search.pm     |  38 ++++++++
 lib/PublicInbox/SearchView.pm |   5 +-
 lib/PublicInbox/WWW.pm        |  14 +++
 lib/PublicInbox/WwwStream.pm  |   4 +-
 lib/PublicInbox/WwwText.pm    | 201 ++++++++++++++++++++++++++++++++++++++++++
 t/psgi_text.t                 |  39 ++++++++
 7 files changed, 299 insertions(+), 4 deletions(-)
 create mode 100644 lib/PublicInbox/WwwText.pm
 create mode 100644 t/psgi_text.t

-- 
EW

[PATCH 1/2] www: implement generic help text

[Eric Wong]

Begin documenting some basic help functionality.
This will probably be edited/tweaked heavily.

I may tweak the anchor names of the HTML thread overview
to be more consistent with each other (old ones will be
supported), so I'm not documenting those, for now...
---
 MANIFEST                     |   2 +
 lib/PublicInbox/Search.pm    |  38 ++++++++
 lib/PublicInbox/WWW.pm       |  14 +++
 lib/PublicInbox/WwwStream.pm |   4 +-
 lib/PublicInbox/WwwText.pm   | 201 +++++++++++++++++++++++++++++++++++++++++++
 t/psgi_text.t                |  39 +++++++++
 6 files changed, 297 insertions(+), 1 deletion(-)
 create mode 100644 lib/PublicInbox/WwwText.pm
 create mode 100644 t/psgi_text.t

diff --git a/MANIFEST b/MANIFEST
index bed6050..306945a 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -82,6 +82,7 @@ lib/PublicInbox/WWW.pm
 lib/PublicInbox/WatchMaildir.pm
 lib/PublicInbox/WwwAttach.pm
 lib/PublicInbox/WwwStream.pm
+lib/PublicInbox/WwwText.pm
 sa_config/Makefile
 sa_config/README
 sa_config/root/etc/spamassassin/public-inbox.pre
@@ -141,6 +142,7 @@ t/plack.t
 t/precheck.t
 t/psgi_attach.t
 t/psgi_mount.t
+t/psgi_text.t
 t/qspawn.t
 t/search.t
 t/spamcheck_spamc.t
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 7561ef4..445c2d8 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -51,6 +51,7 @@ my %bool_pfx_internal = (
 	thread => 'G', # newsGroup (or similar entity - e.g. a web forum name)
 );
 
+# do we still need these? probably not..
 my %bool_pfx_external = (
 	path => 'XPATH',
 	mid => 'Q', # uniQue id (Message-ID)
@@ -62,6 +63,29 @@ my %prob_prefix = (
 	m => 'Q', # 'mid' is exact, 'm' can do partial
 );
 
+# not documenting m: and mid: for now, the using the URLs works w/o Xapian
+our @HELP = (
+	's:' => <<EOF,
+match within Subject only  e.g. s:"a quick brown fox"
+This is a probabilistic search with support for stemming
+and wildcards '*'
+EOF
+	'd:' => <<EOF,
+date range as YYYYMMDD  e.g. d:19931002..20101002
+Open-ended ranges such as d:19931002.. and d:..20101002
+are also supported.
+EOF
+);
+# TODO: (from mairix, some of these are maybe)
+# b (body), f (From:), c (Cc:), n (attachment), t (To:)
+# tc (To:+Cc:), bs (body + Subject), tcf (To: +Cc: +From:)
+#
+# Non-mairix:
+# df (filenames from diff)
+# nq (non-quoted body)
+# da (diff a/ removed lines)
+# db (diff b/ added lines)
+
 my %all_pfx = (%bool_pfx_internal, %bool_pfx_external, %prob_prefix);
 
 sub xpfx { $all_pfx{$_[0]} }
@@ -190,10 +214,14 @@ sub qp {
 	# we do not actually create AltId objects,
 	# just parse the spec to avoid the extra DB handles for now.
 	if (my $altid = $self->{altid}) {
+		my $user_pfx = $self->{-user_pfx} ||= [];
 		for (@$altid) {
 			# $_ = 'serial:gmane:/path/to/gmane.msgmap.sqlite3'
 			/\Aserial:(\w+):/ or next;
 			my $pfx = $1;
+			push @$user_pfx, "$pfx:", <<EOF;
+alternate serial number  e.g. $pfx:12345
+EOF
 			# gmane => XGMANE
 			$qp->add_boolean_prefix($pfx, 'X'.uc($pfx));
 		}
@@ -321,4 +349,14 @@ sub enquire {
 	$self->{enquire} ||= Search::Xapian::Enquire->new($self->{xdb});
 }
 
+sub help {
+	my ($self) = @_;
+	$self->qp; # parse altids
+	my @ret = @HELP;
+	if (my $user_pfx = $self->{-user_pfx}) {
+		push @ret, @$user_pfx;
+	}
+	\@ret;
+}
+
 1;
diff --git a/lib/PublicInbox/WWW.pm b/lib/PublicInbox/WWW.pm
index 6f6a003..4d599fc 100644
--- a/lib/PublicInbox/WWW.pm
+++ b/lib/PublicInbox/WWW.pm
@@ -96,6 +96,8 @@ sub call {
 
 	} elsif ($path_info =~ m!$INBOX_RE/$MID_RE/f/?\z!o) {
 		r301($ctx, $1, $2);
+	} elsif ($path_info =~ m!$INBOX_RE/_/text(?:/(.*))?\z!o) {
+		get_text($ctx, $1, $2);
 
 	# convenience redirects order matters
 	} elsif ($path_info =~ m!$INBOX_RE/([^/]{2,})\z!o) {
@@ -238,6 +240,18 @@ sub get_thread {
 	PublicInbox::View::thread_html($ctx);
 }
 
+# /$INBOX/_/text/$KEY/
+# /$INBOX/_/text/$KEY/raw
+# KEY may contain slashes
+sub get_text {
+	my ($ctx, $inbox, $key) = @_;
+	my $r404 = invalid_inbox($ctx, $inbox);
+	return $r404 if $r404;
+
+	require PublicInbox::WwwText;
+	PublicInbox::WwwText::get_text($ctx, $key);
+}
+
 sub ctx_get {
 	my ($ctx, $key) = @_;
 	my $val = $ctx->{$key};
diff --git a/lib/PublicInbox/WwwStream.pm b/lib/PublicInbox/WwwStream.pm
index 9ed25e1..c89e6de 100644
--- a/lib/PublicInbox/WwwStream.pm
+++ b/lib/PublicInbox/WwwStream.pm
@@ -31,10 +31,12 @@ sub _html_top ($) {
 	my $desc = ascii_html($obj->description);
 	my $title = $ctx->{-title_html} || $desc;
 	my $upfx = $ctx->{-upfx} || '';
+	my $help = $upfx.'_/text/help';
 	my $atom = $ctx->{-atom} || $upfx.'new.atom';
 	my $tip = $ctx->{-html_tip} || '';
 	my $top = "<b>$desc</b>";
-	my $links = "<a\nhref=\"$atom\">Atom feed</a>";
+	my $links = "<a\nhref=\"$help\">help</a> / ".
+			"<a\nhref=\"$atom\">Atom feed</a>";
 	if ($obj->search) {
 		my $q_val = $ctx->{-q_value_html};
 		if (defined $q_val && $q_val ne '') {
diff --git a/lib/PublicInbox/WwwText.pm b/lib/PublicInbox/WwwText.pm
new file mode 100644
index 0000000..ef3446b
--- /dev/null
+++ b/lib/PublicInbox/WwwText.pm
@@ -0,0 +1,201 @@
+# Copyright (C) 2016 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+#
+# serves the /$INBOX/_/* endpoints from :text/* of the git tree
+package PublicInbox::WwwText;
+use strict;
+use warnings;
+use PublicInbox::Linkify;
+use PublicInbox::WwwStream;
+use PublicInbox::Hval qw(ascii_html);
+our $QP_URL = 'https://xapian.org/docs/queryparser.html';
+our $WIKI_URL = 'https://en.wikipedia.org/wiki';
+
+# /$INBOX/_/text/$KEY/ # KEY may contain slashes
+# For now, "help" is the only supported $KEY
+sub get_text {
+	my ($ctx, $key) = @_;
+	my $code = 200;
+
+	$key = 'help' if !defined $key; # this 302s to _/text/help/
+
+	# get the raw text the same way we get mboxrds
+	my $raw = ($key =~ s!/raw\z!!);
+	my $have_tslash = ($key =~ s!/\z!!) if !$raw;
+
+	my $txt = '';
+	if (!_default_text($ctx, $key, \$txt)) {
+		$code = 404;
+		$txt = "404 Not Found ($key)\n";
+	}
+	if ($raw) {
+		return [ $code, [ 'Content-Type', 'text/plain',
+				  'Content-Length', bytes::length($txt) ],
+			[ $txt ] ]
+	}
+
+	# enforce trailing slash for "wget -r" compatibility
+	if (!$have_tslash && $code == 200) {
+		my $url = $ctx->{-inbox}->base_url($ctx->{env});
+		$url .= "_/text/$key/";
+
+		return [ 302, [ 'Content-Type', 'text/plain',
+				'Location', $url ],
+			[ "Redirecting to $url\n" ] ];
+	}
+
+	# Follow git commit message conventions,
+	# first line is the Subject/title
+	my ($title) = ($txt =~ /\A([^\n]*)/s);
+	_do_linkify($txt);
+	$ctx->{-title_html} = ascii_html($title);
+
+	my $nslash = ($key =~ tr!/!/!);
+	$ctx->{-upfx} = '../../../' . ('../' x $nslash);
+
+	PublicInbox::WwwStream->response($ctx, $code, sub {
+		my ($nr, undef) = @_;
+		$nr == 1 ? '<pre>'.$txt.'</pre>' : undef
+	});
+}
+
+sub _do_linkify {
+	my $l = PublicInbox::Linkify->new;
+	$_[0] = $l->linkify_2(ascii_html($l->linkify_1($_[0])));
+}
+
+sub _srch_prefix ($$) {
+	my ($srch, $txt) = @_;
+	my $pad = 0;
+	my $htxt = '';
+	my $help = $srch->help;
+	my $i;
+	for ($i = 0; $i < @$help; $i += 2) {
+		my $pfx = $help->[$i];
+		my $n = length($pfx);
+		$pad = $n if $n > $pad;
+		$htxt .= $pfx . "\0";
+		$htxt .= $help->[$i + 1];
+		$htxt .= "\f\n";
+	}
+	$pad += 2;
+	my $padding = ' ' x ($pad + 8);
+	$htxt =~ s/^/$padding/gms;
+	$htxt =~ s/^$padding(\S+)\0/"        $1".
+				(' ' x ($pad - length($1)))/egms;
+	$htxt =~ s/\f\n/\n/gs;
+	$$txt .= $htxt;
+	1;
+}
+
+
+sub _default_text ($$$) {
+	my ($ctx, $key, $txt) = @_;
+	return if $key ne 'help'; # TODO more keys?
+
+	my $ibx = $ctx->{-inbox};
+	my $base_url = $ibx->base_url($ctx->{env});
+	$$txt .= "public-inbox help for $base_url\n";
+	$$txt .= <<EOF;
+
+overview
+--------
+
+    public-inbox uses Message-ID identifiers in URLs.
+    One may look up messages by substituting Message-IDs
+    (without the leading '<' or trailing '>') into the URL.
+    Forward slash ('/') characters in the Message-IDs
+    need to be escaped as "%2F" (without quotes).
+
+    Thus, it is possible to retrieve any message by its
+    Message-ID by going to:
+
+	$base_url<Message-ID>/
+
+	(without the '<' or '>')
+
+    Message-IDs are described at:
+
+	$WIKI_URL/Message-ID
+
+EOF
+
+	# n.b. we use the Xapian DB for any regeneratable,
+	# order-of-arrival-independent data.
+	my $srch = $ibx->search;
+	if ($srch) {
+		$$txt .= <<EOF;
+search
+------
+
+    This public-inbox has search functionality provided by Xapian.
+
+    It supports typical AND, OR, NOT, '+', '-' queries present
+    in other search engines.
+
+    We also support search prefixes to limit the scope of the
+    search to certain fields using prefixes.
+
+    Prefixes supported in this installation include:
+
+EOF
+		_srch_prefix($srch, $txt);
+
+		$$txt .= <<EOF;
+    The upstream Xapian query parser documentation fully explains
+    the query syntax:
+
+	$QP_URL
+
+message threading
+-----------------
+
+    Message threading is enabled for this public-inbox,
+    additional endpoints for message threads are available:
+
+    * $base_url<Message-ID>/T/#u
+
+      Loads the thread belonging to the given <Message-ID>
+      in flat chronological order.  The "#u" anchor
+      focuses the browser on the given <Message-ID>.
+
+    * $base_url<Message-ID>/t/#u
+
+      Loads the thread belonging to the given <Message-ID>
+      in threaded order with nesting.  For deep threads,
+      this requires a wide display or horizontal scrolling.
+
+    Both of these HTML endpoints are suitable for offline reading
+    using the thread overview at the bottom of each page.
+
+
+    Users of feed readers may follow a particular thread using
+    the Atom feed:
+
+    * $base_url<Message-ID>/t.atom
+
+	https://tools.ietf.org/html/rfc4287
+	$WIKI_URL/Atom_(standard)
+
+    Finally, the gzipped mbox for a thread is available for
+    downloading and importing into your favorite mail client:
+
+    * $base_url<Message-ID>/t.mbox.gz
+
+      We use the mboxrd variant of the mbox format described at:
+
+	$WIKI_URL/Mbox
+
+contact
+-------
+
+    This help text is maintained by public-inbox developers
+    reachable via plain-text email at: meta\@public-inbox.org
+
+EOF
+	# TODO: support admin contact info in ~/.public-inbox/config
+	}
+	1;
+}
+
+1;
diff --git a/t/psgi_text.t b/t/psgi_text.t
new file mode 100644
index 0000000..bf565f8
--- /dev/null
+++ b/t/psgi_text.t
@@ -0,0 +1,39 @@
+# Copyright (C) 2016 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use warnings;
+use Test::More;
+use Email::MIME;
+use File::Temp qw/tempdir/;
+my $tmpdir = tempdir('psgi-text-XXXXXX', TMPDIR => 1, CLEANUP => 1);
+my $maindir = "$tmpdir/main.git";
+my $addr = 'test-public@example.com';
+my $cfgpfx = "publicinbox.test";
+my @mods = qw(HTTP::Request::Common Plack::Test URI::Escape);
+foreach my $mod (@mods) {
+	eval "require $mod";
+	plan skip_all => "$mod missing for psgi_text.t" if $@;
+}
+use_ok $_ foreach @mods;
+use PublicInbox::Import;
+use PublicInbox::Git;
+use PublicInbox::Config;
+use PublicInbox::WWW;
+use_ok 'PublicInbox::WwwText';
+use Plack::Builder;
+my $config = PublicInbox::Config->new({
+	"$cfgpfx.address" => $addr,
+	"$cfgpfx.mainrepo" => $maindir,
+});
+is(0, system(qw(git init -q --bare), $maindir), "git init (main)");
+my $www = PublicInbox::WWW->new($config);
+
+test_psgi(sub { $www->call(@_) }, sub {
+	my ($cb) = @_;
+	my $res;
+	$res = $cb->(GET('/test/_/text/help/'));
+	like($res->content, qr!<title>public-inbox help.*</title>!,
+		'default help');
+});
+
+done_testing();
-- 
EW

[PATCH 2/2] searchview: link to internal help text

[Eric Wong]

The internal help text links to the Xapian query parser
documentation anyways, but also provides information
on which prefixes exist.
---
 lib/PublicInbox/SearchView.pm | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/SearchView.pm b/lib/PublicInbox/SearchView.pm
index 3623a78..4f0811a 100644
--- a/lib/PublicInbox/SearchView.pm
+++ b/lib/PublicInbox/SearchView.pm
@@ -85,12 +85,11 @@ sub mset_summary {
 
 sub err_txt {
 	my ($ctx, $err) = @_;
-	my $u = '//xapian.org/docs/queryparser.html';
-	$u = PublicInbox::Hval::prurl($ctx->{env}, $u);
+	my $u = $ctx->{-inbox}->base_url($ctx->{env}) . '_/text/help/';
 	$err =~ s/^\s*Exception:\s*//; # bad word to show users :P
 	$err = ascii_html($err);
 	"\nBad query: <b>$err</b>\n" .
-		qq{See <a\nhref="$u">$u</a> for Xapian query syntax};
+		qq{See <a\nhref="$u">$u</a> for help on using search};
 }
 
 sub search_nav_top {
-- 
EW

Re: [PATCH 0/2] begin online help for each public-inbox

[Philip Oakley]

From: "Eric Wong" <e@80x24.org>
To: <meta@public-inbox.org>
Cc: "Philip Oakley" <philipoakley@iee.org>
Sent: Thursday, August 18, 2016 10:51 AM
Subject: [PATCH 0/2] begin online help for each public-inbox


>I expect this will be tweaked and edited heavily in coming
> weeks, but this at least a start.

Eric, was this also meant to go to the list?

>
> I'll be keeping everything within the constraints of a
> plain-text email, so no cheating with styling or #fragment
> tags, and definitely no deceptive hiding of URLs to external
> resources.
>
> Fortunately, deploying changes are cheap and relatively
> risk-free with zero user-visible downtime
> (which also needs to be documented :x)
>
> Thanks to Philip Oakley at the git list for reminding me the
> design_www document is actually helpful to users, not just
> hackers :)
>
> 
> https://public-inbox.org/git/D5A2E231FFE74D1C891A1653E1C2D797@PhilipOakley/
>
> Eric Wong (2):
>  www: implement generic help text
>  searchview: link to internal help text
>
> MANIFEST                      |   2 +
> lib/PublicInbox/Search.pm     |  38 ++++++++
> lib/PublicInbox/SearchView.pm |   5 +-
> lib/PublicInbox/WWW.pm        |  14 +++
> lib/PublicInbox/WwwStream.pm  |   4 +-
> lib/PublicInbox/WwwText.pm    | 201 
> ++++++++++++++++++++++++++++++++++++++++++
> t/psgi_text.t                 |  39 ++++++++
> 7 files changed, 299 insertions(+), 4 deletions(-)
> create mode 100644 lib/PublicInbox/WwwText.pm
> create mode 100644 t/psgi_text.t
>
> -- 
> EW
> 

Re: [PATCH 0/2] begin online help for each public-inbox

[Eric Wong]

Philip Oakley <philipoakley@iee.org> wrote:
> From: "Eric Wong" <e@80x24.org>
> To: <meta@public-inbox.org>
> Cc: "Philip Oakley" <philipoakley@iee.org>
> Sent: Thursday, August 18, 2016 10:51 AM
> Subject: [PATCH 0/2] begin online help for each public-inbox
> 
> 
> >I expect this will be tweaked and edited heavily in coming
> >weeks, but this at least a start.
> 
> Eric, was this also meant to go to the list?

Yes, meta@public-inbox.org is the list for public-inbox.
Did you mean git@vger? no; I just Cc:-ed you since you
brought it up in there :)

Favicon for (each) public-inbox

[Philip Oakley]

Eric,

Would it be possible for the Public-Inbox to provide a simple Favicon(*), 
either for the whole site, or in someway for each in-box?

I've added the git public-inbox to my firefox bookmarks as "GitPI" (neat 
eh.), but all I get for the Icon part is a dull grey world. A bit of colour 
would be nice.
--
Philip

(*) I didn't even know what they were called until I searched to see if I 
could customise the Icon. The Mozilla help pages tells me 'no', it comes 
from the source...  https://support.mozilla.org/en-US/questions/1097942 

Re: Favicon for (each) public-inbox

[Eric Wong]

Philip Oakley <philipoakley@iee.org> wrote:
> Eric,
> 
> Would it be possible for the Public-Inbox to provide a simple
> Favicon(*), either for the whole site, or in someway for each
> in-box?

For people running the PSGI themselves, yes, it should 404 so
Plack::App::Cascade (or similar) may be used to fallback to
serving as a static file.

On my server, I have an empty file in favicon.ico so it is
served as a 200 (using Plack::App::File or similar).  This
reduces HTTP requests since favicon-requesting clients tend to
try repeatedly on 404s but not 200s.

> I've added the git public-inbox to my firefox bookmarks as "GitPI"
> (neat eh.), but all I get for the Icon part is a dull grey world. A
> bit of colour would be nice.

I'm afraid having any sort of icon for public-inbox itself would
be a bad representation of the project itself which strives to
be graphics-free and discourage the use of graphics (especially
for branding/marketing).

It seems every image rendering library has been affected by
nasty security bugs in recent years and I'm trying to get users
to reduce their attack surface (and save bandwidth!) by using
non-graphical browsers.

Fwiw, my slow response has been the result of having poor
connectivity the past few days.

> --
> Philip
> 
> (*) I didn't even know what they were called until I searched to see
> if I could customise the Icon. The Mozilla help pages tells me 'no',
> it comes from the source...
> https://support.mozilla.org/en-US/questions/1097942

Ideally, you could patch/hack your browser easily.  I will do my
best to make the public-inbox code easy-to-hack/customize/run,
at least.  I'm extremely picky about UI; mainly, it's _user_
interface, and should be decided for and by a particular user,
not some UI developer.

I've managed without icons for years.  As somebody with bad
eyesight and hearing, I find touch + muscle memory (keystrokes)
the best UI.