* [PATCH 1/5] wwwhighlight: do linkification, too
2019-04-24 23:02 [PATCH 0/5] cgit + WwwHighlight.pm integration Eric Wong
@ 2019-04-24 23:02 ` Eric Wong
2019-04-24 23:02 ` [PATCH 2/5] examples/cgit-commit-filter: remove unused variable Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-04-24 23:02 UTC (permalink / raw)
To: meta
Otherwise, there's no reason to use this API over highlight(1).
Maybe this can be an option in the future; but I'm struggling to
find a reason to not do it by default.
---
lib/PublicInbox/WwwHighlight.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/PublicInbox/WwwHighlight.pm b/lib/PublicInbox/WwwHighlight.pm
index 0191640..d8101b7 100644
--- a/lib/PublicInbox/WwwHighlight.pm
+++ b/lib/PublicInbox/WwwHighlight.pm
@@ -24,6 +24,7 @@ use warnings;
use bytes (); # only for bytes::length
use HTTP::Status qw(status_message);
use parent qw(PublicInbox::HlMod);
+use PublicInbox::Linkify qw();
# TODO: support highlight(1) for distros which don't package the
# SWIG extension. Also, there may be admins who don't want to
@@ -64,7 +65,10 @@ sub call {
return r(405) if $req_method ne 'PUT';
my $bref = read_in_full($env) or return r(500);
+ my $l = PublicInbox::Linkify->new;
+ $l->linkify_1($$bref);
$bref = $self->do_hl($bref, $env->{PATH_INFO});
+ $l->linkify_2($$bref);
my $h = [ 'Content-Type', 'text/html; charset=UTF-8' ];
push @$h, 'Content-Length', bytes::length($$bref);
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/5] examples/cgit-commit-filter: remove unused variable
2019-04-24 23:02 [PATCH 0/5] cgit + WwwHighlight.pm integration Eric Wong
2019-04-24 23:02 ` [PATCH 1/5] wwwhighlight: do linkification, too Eric Wong
@ 2019-04-24 23:02 ` Eric Wong
2019-04-24 23:02 ` [PATCH 3/5] wwwhighlight: deal with no-op highlight Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-04-24 23:02 UTC (permalink / raw)
To: meta
---
examples/cgit-commit-filter.lua | 1 -
1 file changed, 1 deletion(-)
diff --git a/examples/cgit-commit-filter.lua b/examples/cgit-commit-filter.lua
index 7799bef..a8f9e1d 100644
--- a/examples/cgit-commit-filter.lua
+++ b/examples/cgit-commit-filter.lua
@@ -17,7 +17,6 @@ urls['public-inbox.git'] = 'https://public-inbox.org/meta/'
function filter_open(...)
lineno = 0
buffer = ""
- subject = ""
end
function filter_close()
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/5] wwwhighlight: deal with no-op highlight
2019-04-24 23:02 [PATCH 0/5] cgit + WwwHighlight.pm integration Eric Wong
2019-04-24 23:02 ` [PATCH 1/5] wwwhighlight: do linkification, too Eric Wong
2019-04-24 23:02 ` [PATCH 2/5] examples/cgit-commit-filter: remove unused variable Eric Wong
@ 2019-04-24 23:02 ` Eric Wong
2019-04-24 23:02 ` [PATCH 4/5] examples: cgit filter for use with WwwHighlight Eric Wong
2019-04-24 23:02 ` [PATCH 5/5] examples/cgit-commit-filter.lua: some doc updates Eric Wong
4 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-04-24 23:02 UTC (permalink / raw)
To: meta
Not all inputs are highlight-able, so reuse the original
input and just linkify it if it can't be highlighted.
---
lib/PublicInbox/WwwHighlight.pm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/WwwHighlight.pm b/lib/PublicInbox/WwwHighlight.pm
index d8101b7..bc349f8 100644
--- a/lib/PublicInbox/WwwHighlight.pm
+++ b/lib/PublicInbox/WwwHighlight.pm
@@ -25,6 +25,7 @@ use bytes (); # only for bytes::length
use HTTP::Status qw(status_message);
use parent qw(PublicInbox::HlMod);
use PublicInbox::Linkify qw();
+use PublicInbox::Hval qw(ascii_html);
# TODO: support highlight(1) for distros which don't package the
# SWIG extension. Also, there may be admins who don't want to
@@ -67,7 +68,11 @@ sub call {
my $bref = read_in_full($env) or return r(500);
my $l = PublicInbox::Linkify->new;
$l->linkify_1($$bref);
- $bref = $self->do_hl($bref, $env->{PATH_INFO});
+ if (my $res = $self->do_hl($bref, $env->{PATH_INFO})) {
+ $bref = $res;
+ } else {
+ $$bref = ascii_html($$bref);
+ }
$l->linkify_2($$bref);
my $h = [ 'Content-Type', 'text/html; charset=UTF-8' ];
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 4/5] examples: cgit filter for use with WwwHighlight
2019-04-24 23:02 [PATCH 0/5] cgit + WwwHighlight.pm integration Eric Wong
` (2 preceding siblings ...)
2019-04-24 23:02 ` [PATCH 3/5] wwwhighlight: deal with no-op highlight Eric Wong
@ 2019-04-24 23:02 ` Eric Wong
2019-04-25 3:19 ` [PATCH v2] " Eric Wong
2019-04-24 23:02 ` [PATCH 5/5] examples/cgit-commit-filter.lua: some doc updates Eric Wong
4 siblings, 1 reply; 8+ messages in thread
From: Eric Wong @ 2019-04-24 23:02 UTC (permalink / raw)
To: meta
I'm using this as the cgit about-filter and source-filter
in https://80x24.org/public-inbox.git
---
MANIFEST | 1 +
examples/cgit-wwwhighlight-filter.lua | 57 +++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
create mode 100644 examples/cgit-wwwhighlight-filter.lua
diff --git a/MANIFEST b/MANIFEST
index 9858b9e..ed8ff49 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -39,6 +39,7 @@ examples/apache2_perl.conf
examples/apache2_perl_old.conf
examples/cgi-webrick.rb
examples/cgit-commit-filter.lua
+examples/cgit-wwwhighlight-filter.lua
examples/cgit.psgi
examples/highlight.psgi
examples/logrotate.conf
diff --git a/examples/cgit-wwwhighlight-filter.lua b/examples/cgit-wwwhighlight-filter.lua
new file mode 100644
index 0000000..dafb7a8
--- /dev/null
+++ b/examples/cgit-wwwhighlight-filter.lua
@@ -0,0 +1,57 @@
+-- Copyright (C) 2019 all contributors <meta@public-inbox.org>
+-- License: GPL-2.0+ <https://www.gnu.org/licenses/gpl-2.0.txt>
+--
+-- This filter accesses the PublicInbox::WwwHighlight PSGI endpoint
+-- (see examples/highlight.psgi)
+--
+-- Dependencies: lua-http
+--
+-- disclaimer: written by someone who does not know Lua.
+--
+-- This requires cgit linked with Lua
+-- Usage (in your cgitrc(5) config file):
+--
+-- source-filter=lua:/path/to/this/script.lua
+-- about-filter=lua:/path/to/this/script.lua
+--
+local wwwhighlight_url = 'http://127.0.0.1:9090/'
+local req_timeout = 10
+
+-- this is nil when used as about-filter
+local fn
+
+function filter_open(...)
+ req_body = ""
+ fn = select(1, ...)
+ if fn then
+ local http_util = require 'http.util'
+ wwwhighlight_url = wwwhighlight_url .. http_util.encodeURI(fn)
+ end
+end
+
+-- buffer the entire source in memory
+function filter_write(str)
+ req_body = req_body .. str
+end
+
+function filter_close()
+ local request = require 'http.request'
+ local req = request.new_from_uri(wwwhighlight_url)
+ req.headers:upsert(':method', 'PUT')
+ req:set_body(req_body)
+ local headers, stream = req:go(req_timeout)
+ local body, err = stream:get_body_as_string()
+ if fn then
+ html("<pre>")
+ end
+ if not body and err then
+ io.stderr:write(tostring(err), "\n")
+ html_txt(req_body)
+ else
+ html(body)
+ end
+ if fn then
+ html("</pre>")
+ end
+ return 0
+end
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v2] examples: cgit filter for use with WwwHighlight
2019-04-24 23:02 ` [PATCH 4/5] examples: cgit filter for use with WwwHighlight Eric Wong
@ 2019-04-25 3:19 ` Eric Wong
2019-04-25 3:25 ` Eric Wong
0 siblings, 1 reply; 8+ messages in thread
From: Eric Wong @ 2019-04-25 3:19 UTC (permalink / raw)
To: meta
I'm using this as the cgit about-filter and source-filter
in https://80x24.org/public-inbox.git
---
v2:
* proper error handling and behavior w.r.t. large files
* do not send Expect: header (and waiting for 100-continue)
* correctly distinguish between about-filter and source-filter
(v1 was creating nested <pre> when used as source-filter)
interdiff:
$ U=https://public-inbox.org/meta
$ interdiff <(curl -s $U/dafb7a8/s/?b=x.lua) <(curl -s $U/a267d1c/s/?b=x.lua)
MANIFEST | 1 +
examples/cgit-wwwhighlight-filter.lua | 105 ++++++++++++++++++++++++++
2 files changed, 106 insertions(+)
create mode 100644 examples/cgit-wwwhighlight-filter.lua
diff --git a/MANIFEST b/MANIFEST
index 9858b9e..ed8ff49 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -39,6 +39,7 @@ examples/apache2_perl.conf
examples/apache2_perl_old.conf
examples/cgi-webrick.rb
examples/cgit-commit-filter.lua
+examples/cgit-wwwhighlight-filter.lua
examples/cgit.psgi
examples/highlight.psgi
examples/logrotate.conf
diff --git a/examples/cgit-wwwhighlight-filter.lua b/examples/cgit-wwwhighlight-filter.lua
new file mode 100644
index 0000000..a267d1c
--- /dev/null
+++ b/examples/cgit-wwwhighlight-filter.lua
@@ -0,0 +1,105 @@
+-- Copyright (C) 2019 all contributors <meta@public-inbox.org>
+-- License: GPL-2.0+ <https://www.gnu.org/licenses/gpl-2.0.txt>
+--
+-- This filter accesses the PublicInbox::WwwHighlight PSGI endpoint
+-- (see examples/highlight.psgi)
+--
+-- Dependencies: lua-http
+--
+-- disclaimer: written by someone who does not know Lua.
+--
+-- This requires cgit linked with Lua
+-- Usage (in your cgitrc(5) config file):
+--
+-- source-filter=lua:/path/to/this/script.lua
+-- about-filter=lua:/path/to/this/script.lua
+--
+local wwwhighlight_url = 'http://127.0.0.1:9090/'
+local req_timeout = 10
+local too_big = false
+
+-- match $PublicInbox::HTTP::MAX_REQUEST_BUFFER
+local max_len = 10 * 1024 * 1024
+
+-- about-filter needs surrounding <pre> tags if all we do is
+-- highlight and linkify
+local pre = true
+
+function filter_open(...)
+ req_body = ""
+
+ -- detect when we're used in an about-filter
+ local repo_url = os.getenv('CGIT_REPO_URL')
+ if repo_url then
+ local path_info = os.getenv('PATH_INFO')
+ rurl = path_info:match("^/(.+)/about/?$")
+ pre = rurl == repo_url
+ end
+
+ -- hand filename off for language detection
+ local fn = select(1, ...)
+ if fn then
+ local http_util = require 'http.util'
+ wwwhighlight_url = wwwhighlight_url .. http_util.encodeURI(fn)
+ end
+end
+
+-- try to buffer the entire source in memory
+function filter_write(str)
+ if too_big then
+ html(str)
+ elseif (req_body:len() + str:len()) > max_len then
+ too_big = true
+ req_body = ""
+ html(req_body)
+ html(str)
+ else
+ req_body = req_body .. str
+ end
+end
+
+function fail(err)
+ io.stderr:write(tostring(err), "\n")
+ if pre then
+ html("<pre>")
+ end
+ html_txt(req_body)
+ if pre then
+ html("</pre>")
+ end
+ return 1
+end
+
+function filter_close()
+ if too_big then
+ return 0
+ end
+ local request = require 'http.request'
+ local req = request.new_from_uri(wwwhighlight_url)
+ req.headers:upsert(':method', 'PUT')
+ req:set_body(req_body)
+
+ -- don't wait for 100-Continue message from the PSGI app
+ req.headers:delete('expect')
+
+ local headers, stream = req:go(req_timeout)
+ if headers == nil then
+ return fail(stream)
+ end
+ local status = headers:get(':status')
+ if status ~= '200' then
+ return fail('status ' .. status)
+ end
+ local body, err = stream:get_body_as_string()
+ if not body and err then
+ return fail(err)
+ end
+ if pre then
+ html("<pre>")
+ end
+ html(body)
+ if pre then
+ html("</pre>")
+ end
+ return 0
+end
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 5/5] examples/cgit-commit-filter.lua: some doc updates
2019-04-24 23:02 [PATCH 0/5] cgit + WwwHighlight.pm integration Eric Wong
` (3 preceding siblings ...)
2019-04-24 23:02 ` [PATCH 4/5] examples: cgit filter for use with WwwHighlight Eric Wong
@ 2019-04-24 23:02 ` Eric Wong
4 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-04-24 23:02 UTC (permalink / raw)
To: meta
It's been a while since I wrote this, and it needs to be kept
up-to-date with some advances in our Perl code.
---
examples/cgit-commit-filter.lua | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/examples/cgit-commit-filter.lua b/examples/cgit-commit-filter.lua
index a8f9e1d..1677253 100644
--- a/examples/cgit-commit-filter.lua
+++ b/examples/cgit-commit-filter.lua
@@ -13,6 +13,9 @@
local urls = {}
urls['public-inbox.git'] = 'https://public-inbox.org/meta/'
-- additional URLs here...
+-- TODO we should be able to auto-generate this based on "coderepo"
+-- directives in the public-inbox config file; but keep in mind
+-- the mapping is M:N between inboxes and coderepos
function filter_open(...)
lineno = 0
@@ -20,6 +23,9 @@ function filter_open(...)
end
function filter_close()
+ -- cgit opens and closes this filter for the commit subject
+ -- and body separately, and we only generate the link based
+ -- on the commit subject:
if lineno == 1 and string.find(buffer, "\n") == nil then
u = urls[os.getenv('CGIT_REPO_URL')]
if u == nil then
@@ -32,6 +38,9 @@ function filter_close()
html('</tt></a>')
end
else
+ -- pass the body-through as-is
+ -- TODO: optionally use WwwHighlight for linkification like
+ -- cgit-wwwhighlight-filter.lua
html(buffer)
end
return 0
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread