* [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
@ 2019-05-09 7:50 ` Eric Wong
2019-05-09 7:50 ` [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header Eric Wong
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:50 UTC (permalink / raw)
To: meta
Fortunately, there is a pattern to most of these package names
in all distros I've tested (and hopefully other BSDs have them,
too).
Then, reorder the INSTALL document to mention the top-level
modules, first, so users can avoid needing to type extra
dependencies. However, we still list some implicit dependencies
in case the upstream package drops dependencies independently of
us.
Finally, Devel::Peek is not a dependency worth making optional
since it's bundled by Perl upstream. Fedora/RH-based distros
are the only one which turn it into a non-standard package when
Perl5 is installed.
---
INSTALL | 197 ++++++++++++++++++++++++++++++++--------------------
Makefile.PL | 8 +++
2 files changed, 128 insertions(+), 77 deletions(-)
diff --git a/INSTALL b/INSTALL
index fafaf57..99836c1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -9,8 +9,8 @@ public-inbox is developed on Debian GNU/Linux systems and will
never depend on packages outside of the "main" component of
the "stable" distribution, currently Debian 9.x ("stretch")
-Most packages are available in other GNU/Linux distributions;
-and FreeBSD support can happen.
+Most packages are available in other GNU/Linux distributions
+and FreeBSD.
TODO: this still needs to be documented better,
also see the scripts/ and sa_config/ directories in the source tree
@@ -23,7 +23,7 @@ functionality. The core tools are, of course:
* Git (1.8.0+, 2.6+ for writing v2 repositories)
* Perl 5.10.1+
-* SQLite (needed for Xapian use)
+* DBD::SQLite (needed for NNTP, message threading, and v2 repositories)
To accept incoming mail into a public inbox, you'll likely want:
@@ -32,88 +32,131 @@ To accept incoming mail into a public inbox, you'll likely want:
Beyond that, there is a long list of Perl modules required, starting with:
-* Date::Parse deb: libtimedate-perl
- rpm: perl-Time-ParseDate
+* Date::Parse deb: libtimedate-perl
+ pkg: p5-TimeDate
+ rpm: perl-TimeDate
-* Email::MIME deb: libemail-mime-perl
- rpm: perl-Email-MIME
+* Devel::Peek deb: libperl5.$MINOR (e.g. libperl5.24)
+ pkg: perl5
+ rpm: perl-Devel-Peek
+ (typically installed alongside Perl5)
-* Email::MIME::ContentType deb: libemail-mime-contenttype-perl
- rpm: perl-Email-MIME-ContentType
+* Email::MIME deb: libemail-mime-perl
+ pkg: p5-Email-MIME
+ rpm: perl-Email-MIME
-* Encode::MIME::Header deb: libencode-perl
- rpm: perl-Encode
+* Encode::MIME::Header deb: libencode-perl
+ pkg: perl5
+ rpm: perl-Encode
-* Plack deb: libplack-perl
- rpm: perl-Plack, perl-Plack-Test,
- (for HTML/Atom generation)
+* Plack deb: libplack-perl
+ pkg: p5-Plack
+ rpm: perl-Plack, perl-Plack-Test,
+ (for HTML/Atom generation)
-* URI::Escape deb: liburi-perl
- rpm: perl-URI
- (for HTML/Atom generation)
+* URI::Escape deb: liburi-perl
+ pkg: p5-URI
+ rpm: perl-URI
+ (for HTML/Atom generation)
-Where "deb" indicates package names for Debian-derived distributions and
+Where "deb" indicates package names for Debian-derived distributions,
+"pkg" is for the FreeBSD package (maybe other common BSDs, too), and
"rpm" is for RPM-based distributions (only known to work on Fedora).
Numerous optional modules are likely to be useful as well:
- - Search::Xapian deb: libsearch-xapian-perl
- rpm: perl-Search-Xapian
- (for NNTP service or gzipped mbox over HTTP)
-
- - IO::Compress::Gzip deb: perl-modules (or libio-compress-perl)
- rpm: perl-PerlIO-gzip
- (for gzipped mbox over HTTP)
-
- - DBI deb: libdbi-perl
- rpm: perl-DBI
- (for gzipped mbox over HTTP)
-
- - DBD::SQLite deb: libdbd-sqlite3-perl
- rpm: perl-DBD-SQLite
- (for NNTP service or gzipped mbox over HTTP)
-
- - Net::Server deb: libnet-server-perl
- rpm: perl-Net-Server
- (for HTTP/NNTP servers as standalone daemons,
- not needed as systemd services)
-
- - Filesys::Notify::Simple deb: libfilesys-notify-simple-perl
- rpm: perl-Filesys-Notify-Simple
- (for public-inbox-watch)
-
- - Inline::C[7] deb: libinline-c-perl
- (speeds up spawning of git(1) on Linux
- (see public-inbox-daemon(8))
-
- - Plack::Middleware::ReverseProxy
-
- deb: libplack-middleware-reverseproxy-perl
- rpm: perl-Plack-Middleware-ReverseProxy
- (ensures redirects are correct when running
- behind nginx or Varnish)
-
- - Plack::Middleware::Deflater
-
- deb: libplack-middleware-deflater-perl
- rpm: perl-Plack-Middleware-Deflater
- (saves bandwidth on responses)
-
- - Socket6 deb: libsocket6-perl
- rpm: perl-Socket6
- (pulled in by SpamAssassin and Net::Server,
- only necessary if using IPv6 with
- Plack::Middleware::AccessLog or similar
- on Perl <= 5.12)
-
- - Crypt::CBC deb: libcrypt-cbc-perl
- (for the rarely-used PublicInbox::Unsubscribe)
-
-On Fedora systems, you'll probably also end up wanting
-perl-Test-HTTP-Server-Simple, perl-Devel-Peek, and perl-IPC-Run to run the
-test suite. On Debian systems, libxml-feed-perl and libipc-run-perl
-will aid in running the test suite (XML::Feed and IPC::Run respectively,
-on CPAN).
+- DBD::SQLite deb: libdbd-sqlite3-perl
+ pkg: p5-DBD-SQLite
+ rpm: perl-DBD-SQLite
+ (for v2, NNTP, or gzipped mboxes)
+
+- Search::Xapian deb: libsearch-xapian-perl
+ pkg: p5-Search-Xapian
+ rpm: perl-Search-Xapian
+ (for v2, HTTP search)
+
+- Net::Server deb: libnet-server-perl
+ pkg: pkg-Net-Server
+ rpm: perl-Net-Server
+ (for HTTP/NNTP background daemons,
+ not needed as systemd services or
+ foreground servers)
+
+- Inline::C[7] deb: libinline-c-perl
+ pkg: pkg-Inline-C
+ (speeds up process spawning on Linux,
+ see public-inbox-daemon(8))
+
+- Plack::Middleware::ReverseProxy deb: libplack-middleware-reverseproxy-perl
+ pkg: p5-Plack-Middleware-ReverseProxy
+ rpm: perl-Plack-Middleware-ReverseProxy
+ (ensures redirects are correct when running
+ behind nginx or Varnish)
+
+- Plack::Middleware::Deflater deb: libplack-middleware-deflater-perl
+ pkg: p5 -Plack-Middleware-Deflater
+ rpm: perl-Plack-Middleware-Deflater
+ (saves bandwidth on responses)
+
+* highlight deb: libhighlight-perl
+ (for syntax highlighting with coderepo)
+
+The following modules are typically pulled in by dependencies listed
+above, so there is no need to explicitly install them:
+
+- Email::MIME::ContentType deb: libemail-mime-contenttype-perl
+ pkg: p5-Email-MIME-ContentType
+ rpm: perl-Email-MIME-ContentType
+ (pulled in by Email::MIME)
+
+- Email::Simple deb: libemail-simple-perl
+ pkg: p5-Email-Simple
+ rpm: perl-Email-Simple
+ (pulled in by Email::MIME)
+
+- DBI deb: libdbi-perl
+ pkg: p5-DBI
+ rpm: perl-DBI
+ (pulled in by DBD::SQLite)
+
+- Filesys::Notify::Simple deb: libfilesys-notify-simple-perl
+ pkg: pkg-Filesys-Notify-Simple
+ rpm: perl-Filesys-Notify-Simple
+ (for public-inbox-watch, pulled in by Plack)
+
+- IO::Compress::Gzip deb: perl-modules (or libio-compress-perl)
+ rpm: perl-PerlIO-gzip
+ pkg: perl5
+ (for gzipped mbox over HTTP)
+
+Uncommonly needed modules:
+
+- Socket6 deb: libsocket6-perl
+ pkg: p5-Socket6
+ rpm: perl-Socket6
+ (pulled in by SpamAssassin and Net::Server,
+ only necessary if using IPv6 with
+ Plack::Middleware::AccessLog or similar
+ on Perl <= 5.12)
+
+- Crypt::CBC deb: libcrypt-cbc-perl
+ pkg: p5-Crypt-CBC
+ (for PublicInbox::Unsubscribe (rarely used))
+
+Optional packages testing and development:
+
+- IPC::Run deb: libipc-run-perl
+ pkg: p5-IPC-Run
+ rpm: perl-IPC-Run
+
+- XML::Feed deb: libxml-feed-perl
+ pkg: p5-XML-Feed
+ rpm: perl-XML-Feed
+
+- Test::HTTP::Server::Simple deb: libtest-http-server-simple-perl
+ pkg: p5-Test-HTTP-Server-Simple
+ rpm: perl-Test-HTTP-Server-Simple
+ (XXX is this really needed?)
standard MakeMaker installation (Perl)
--------------------------------------
@@ -137,8 +180,8 @@ public-inbox will never store unregeneratable data in Xapian
or any other search database we might use; Xapian corruption
will not destroy critical data.
-See the public-inbox-overview(7) man page for the next steps once the
-installation is complete.
+See the public-inbox-overview(7) man page for the next steps once
+the installation is complete.
Copyright
---------
diff --git a/Makefile.PL b/Makefile.PL
index 3bb0072..b857667 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -29,10 +29,18 @@ WriteMakefile(
# Keep this sorted and synced to the INSTALL document
'Date::Parse' => 0,
'Email::MIME' => 0,
+
+ # the following should be pulled in by Email::MIME:
'Email::MIME::ContentType' => 0,
'Email::Simple' => 0,
+
+ # "libencode-perl" on Debian, `perl5' on FreeBSD
'Encode::MIME::Header' => 0,
+ # libperl$PERL_VERSION on Debian, `perl5' on FreeBSD,
+ # but Fedora seems to need this separately
+ 'Devel::Peek' => 0,
+
# TODO: these should really be made optional...
'Plack' => 0,
'URI::Escape' => 0,
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
2019-05-09 7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
@ 2019-05-09 7:50 ` Eric Wong
2019-05-09 7:50 ` [PATCH 3/9] INSTALL: note xapian-compact(1) tool Eric Wong
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:50 UTC (permalink / raw)
To: meta
Email::MIME uses Encode::MIME::Header and depends on that
appropriately; however we depend on other parts of the Encode
distribution, but that's bundled with Perl by upstream, anyways;
and should place no additional burden on users.
---
INSTALL | 9 +++++----
Makefile.PL | 5 +++--
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/INSTALL b/INSTALL
index 99836c1..4574311 100644
--- a/INSTALL
+++ b/INSTALL
@@ -45,10 +45,6 @@ Beyond that, there is a long list of Perl modules required, starting with:
pkg: p5-Email-MIME
rpm: perl-Email-MIME
-* Encode::MIME::Header deb: libencode-perl
- pkg: perl5
- rpm: perl-Encode
-
* Plack deb: libplack-perl
pkg: p5-Plack
rpm: perl-Plack, perl-Plack-Test,
@@ -114,6 +110,11 @@ above, so there is no need to explicitly install them:
rpm: perl-Email-Simple
(pulled in by Email::MIME)
+* Encode deb: libperl5.$MINOR (or libencode-perl)
+ pkg: perl5
+ rpm: perl-Encode
+ (likely installed with Perl)
+
- DBI deb: libdbi-perl
pkg: p5-DBI
rpm: perl-DBI
diff --git a/Makefile.PL b/Makefile.PL
index b857667..eda7319 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -34,8 +34,9 @@ WriteMakefile(
'Email::MIME::ContentType' => 0,
'Email::Simple' => 0,
- # "libencode-perl" on Debian, `perl5' on FreeBSD
- 'Encode::MIME::Header' => 0,
+ # libperl$PERL_VERSION or libencode-perl on Debian,
+ # `perl5' on FreeBSD
+ 'Encode' => 0,
# libperl$PERL_VERSION on Debian, `perl5' on FreeBSD,
# but Fedora seems to need this separately
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 3/9] INSTALL: note xapian-compact(1) tool
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
2019-05-09 7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
2019-05-09 7:50 ` [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header Eric Wong
@ 2019-05-09 7:50 ` Eric Wong
2019-05-09 7:50 ` [PATCH 4/9] t/purge.t: fix unreferenced variable Eric Wong
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:50 UTC (permalink / raw)
To: meta
The Perl Xapian bindings do not support compacting, yet,
so we rely on the xapian-compact tool.
---
INSTALL | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/INSTALL b/INSTALL
index 4574311..313a295 100644
--- a/INSTALL
+++ b/INSTALL
@@ -97,6 +97,11 @@ Numerous optional modules are likely to be useful as well:
* highlight deb: libhighlight-perl
(for syntax highlighting with coderepo)
+* xapian-compact (tool) deb: xapian-tools
+ pkg: xapian-core
+ rpm: xapian-core
+ (optional, for public-inbox-compact(1))
+
The following modules are typically pulled in by dependencies listed
above, so there is no need to explicitly install them:
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 4/9] t/purge.t: fix unreferenced variable
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (2 preceding siblings ...)
2019-05-09 7:50 ` [PATCH 3/9] INSTALL: note xapian-compact(1) tool Eric Wong
@ 2019-05-09 7:50 ` Eric Wong
2019-05-09 7:50 ` [PATCH 5/9] doc: use bullet list for wwwlisting options Eric Wong
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:50 UTC (permalink / raw)
To: meta
---
t/purge.t | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/purge.t b/t/purge.t
index 9406005..b518c26 100644
--- a/t/purge.t
+++ b/t/purge.t
@@ -9,7 +9,7 @@ require_git(2.6);
my @mods = qw(IPC::Run DBI DBD::SQLite Search::Xapian);
foreach my $mod (@mods) {
eval "require $mod";
- plan skip_all => "missing $_ for t/purge.t" if $@;
+ plan skip_all => "missing $mod for t/purge.t" if $@;
};
use Cwd qw(abs_path);
my $purge = abs_path('blib/script/public-inbox-purge');
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 5/9] doc: use bullet list for wwwlisting options
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (3 preceding siblings ...)
2019-05-09 7:50 ` [PATCH 4/9] t/purge.t: fix unreferenced variable Eric Wong
@ 2019-05-09 7:50 ` Eric Wong
2019-05-09 7:51 ` [PATCH 6/9] doc: give txt files proper titles Eric Wong
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:50 UTC (permalink / raw)
To: meta
Otherwise, pod2man complains about "=item 404" not starting
with a letter and thinking it's part of a numbered list.
---
Documentation/public-inbox-config.pod | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod
index d44c8f3..db81bf1 100644
--- a/Documentation/public-inbox-config.pod
+++ b/Documentation/public-inbox-config.pod
@@ -241,19 +241,15 @@ Valid values are:
=over 8
-=item all
+=item * all
+- Show all inboxes
-Show all inboxes
-
-=item 404
-
-Return a 404 page. This is useful to allow customization with
+=item * 404
+- Return a 404 page. This is useful to allow customization with
L<Plack::App::Cascade(3pm)>
-=item match=domain
-
-Only show inboxes with URLs which belong to the domain of the HTTP
-request
+=item * match=domain
+- Only show inboxes with URLs which belong to the domain of the HTTP request
=for TODO comment
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6/9] doc: give txt files proper titles
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (4 preceding siblings ...)
2019-05-09 7:50 ` [PATCH 5/9] doc: use bullet list for wwwlisting options Eric Wong
@ 2019-05-09 7:51 ` Eric Wong
2019-05-09 7:51 ` [PATCH 7/9] Makefile.PL: improve make(1) portability Eric Wong
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:51 UTC (permalink / raw)
To: meta
We can fix the redundant rule in include.mk which causes
make(1) on FreeBSD to complain; but HTML docs will likely
still require GNU make.
---
Documentation/design_www.txt | 2 ++
Documentation/hosted.txt | 2 ++
Documentation/include.mk | 5 +++--
Documentation/txt2pre | 3 +--
4 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/design_www.txt b/Documentation/design_www.txt
index cd715b2..f15a556 100644
--- a/Documentation/design_www.txt
+++ b/Documentation/design_www.txt
@@ -1,3 +1,5 @@
+PublicInbox::WWW (PSGI interface) design notes
+
URL and anchor naming
---------------------
diff --git a/Documentation/hosted.txt b/Documentation/hosted.txt
index 42926d2..0b1fb92 100644
--- a/Documentation/hosted.txt
+++ b/Documentation/hosted.txt
@@ -1,3 +1,5 @@
+unofficially hosted mirrors at public-inbox.org
+
In addition to eating our own dogfood at <https://public-inbox.org/meta/>,
public-inbox.org hosts unofficial archives for several other projects
as a service to their users and to further test our own software.
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 02cbef3..9089e06 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -87,8 +87,9 @@ dtxt := $(addprefix Documentation/, $(dtxt)) $(mantxt)
Documentation/standards.txt : Documentation/standards.perl
$(PERL) $< >$@+ && mv $@+ $@
-%.html: %.txt
- TITLE="$(basename $(<F))" $(txt2pre)
+Documentation/%.html: Documentation/%.txt
+ $(txt2pre)
+
%.html: %
$(txt2pre)
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 4c4b2ca..9a81a24 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -11,8 +11,7 @@ use PublicInbox::Linkify;
use PublicInbox::Hval qw(ascii_html);
my $str = eval { local $/; <> };
-my $title = $ENV{TITLE};
-($title) = ($str =~ /\A([^\n]+)/) unless $title;
+my ($title) = ($str =~ /\A([^\n]+)/);
$title = ascii_html($title);
my $l = PublicInbox::Linkify->new;
$str = $l->linkify_1($str);
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 7/9] Makefile.PL: improve make(1) portability
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (5 preceding siblings ...)
2019-05-09 7:51 ` [PATCH 6/9] doc: give txt files proper titles Eric Wong
@ 2019-05-09 7:51 ` Eric Wong
2019-05-09 7:51 ` [PATCH 8/9] doc/include.mk: remove git use and redundant declarations Eric Wong
2019-05-09 7:51 ` [PATCH 9/9] ci: include some scripts to make CI easier Eric Wong
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:51 UTC (permalink / raw)
To: meta
The `shell' function appears missing, so we'll rely on Bourne
shell expansioN, instead.
Use "$?" instead of "$<" since the latter is only specified
for inference and .DEFAULT rules, not target rules.
Tested on FreeBSD make(1) and bmake(1) on Debian.
---
Makefile.PL | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Makefile.PL b/Makefile.PL
index eda7319..6be913b 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -59,7 +59,7 @@ sub MY::postamble {
EATMYDATA =
-include config.mak
-include Documentation/include.mk
-N ?= \$(shell echo \$\$(( \$\$(nproc 2>/dev/null || echo 2) + 1)))
+N = \$\$(( \$\$(nproc 2>/dev/null || gnproc 2>/dev/null || echo 2) + 1 ))
SCRIPTS := scripts/ssoma-replay
my_syntax := \$(addsuffix .syntax, $PM_FILES \$(EXE_FILES) \$(SCRIPTS))
@@ -70,13 +70,13 @@ my_syntax := \$(addsuffix .syntax, $PM_FILES \$(EXE_FILES) \$(SCRIPTS))
syntax:: \$(my_syntax)
check-manifest :: MANIFEST
- if git ls-files >\$<.gen 2>&1; then diff -u \$< \$<.gen; fi
+ if git ls-files >\$?.gen 2>&1; then diff -u \$? \$?.gen; fi
check:: pure_all check-manifest
\$(EATMYDATA) prove -lv -j\$(N)
lib/PublicInbox/UserContent.pm :: contrib/css/216dark.css
- @\$(PERL) -I lib \$@ \$<
+ \$(PERL) -I lib \$@ \$?
EOF
}
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 8/9] doc/include.mk: remove git use and redundant declarations
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (6 preceding siblings ...)
2019-05-09 7:51 ` [PATCH 7/9] Makefile.PL: improve make(1) portability Eric Wong
@ 2019-05-09 7:51 ` Eric Wong
2019-05-09 7:51 ` [PATCH 9/9] ci: include some scripts to make CI easier Eric Wong
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:51 UTC (permalink / raw)
To: meta
And document that we still have GNU-isms in that
include.mk Makefile (and may continue to do so).
Finally, take advantage of GNU-isms to warn users
to run "gmake" to build all manpages.
---
Documentation/include.mk | 26 +++++++++++++++++++-------
1 file changed, 19 insertions(+), 7 deletions(-)
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 9089e06..2a02611 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -2,10 +2,18 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
all::
+# Note: some GNU-isms present and required to build docs
+# (including manpages), but at least this should not trigger
+# warnings with BSD make(1) when running "make check"
+# Maybe it's not worth it to support non-GNU make, though...
RSYNC = rsync
RSYNC_DEST = public-inbox.org:/srv/public-inbox/
-docs := README COPYING INSTALL TODO HACKING
-docs += $(shell git ls-files 'Documentation/*.txt')
+txt := INSTALL README COPYING TODO HACKING
+dtxt := design_notes.txt design_www.txt dc-dlvr-spam-flow.txt hosted.txt
+dtxt += standards.txt
+dtxt := $(addprefix Documentation/, $(dtxt))
+docs := $(txt) $(dtxt)
+
INSTALL = install
PODMAN = pod2man
PODMAN_OPTS = -v --stderr -d 1993-10-02 -c 'public-inbox user manual'
@@ -71,6 +79,7 @@ manuals += $(m8)
mantxt = $(addprefix Documentation/, $(addsuffix .txt, $(manuals)))
docs += $(mantxt)
+dtxt += $(mantxt)
all :: $(mantxt)
@@ -79,10 +88,6 @@ Documentation/%.txt : Documentation/%.pod
txt2pre = $(PERL) -I lib ./Documentation/txt2pre <$< >$@+ && \
touch -r $< $@+ && mv $@+ $@
-txt := INSTALL README COPYING TODO
-dtxt := design_notes.txt design_www.txt dc-dlvr-spam-flow.txt hosted.txt
-dtxt += standards.txt
-dtxt := $(addprefix Documentation/, $(dtxt)) $(mantxt)
Documentation/standards.txt : Documentation/standards.perl
$(PERL) $< >$@+ && mv $@+ $@
@@ -96,7 +101,10 @@ Documentation/%.html: Documentation/%.txt
docs_html := $(addsuffix .html, $(subst .txt,,$(dtxt)) $(txt))
html: $(docs_html)
gz_docs := $(addsuffix .gz, $(docs) $(docs_html))
-rsync_docs := $(gz_docs) $(docs) $(txt) $(docs_html) $(dtxt)
+rsync_docs := $(gz_docs) $(docs) $(docs_html)
+
+doc: $(docs)
+
%.gz: %
gzip -9 --rsyncable <$< >$@+
touch -r $< $@+
@@ -111,3 +119,7 @@ clean-doc:
$(RM) $(man1) $(man5) $(man7) $(gz_docs) $(docs_html) $(mantxt)
clean :: clean-doc
+
+pure_all ::
+ @if test x"$(addprefix g, make)" != xgmake; then \
+ echo W: gmake is currently required to build manpages; fi
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 9/9] ci: include some scripts to make CI easier
2019-05-09 7:50 [PATCH 0/9] install updates Eric Wong
` (7 preceding siblings ...)
2019-05-09 7:51 ` [PATCH 8/9] doc/include.mk: remove git use and redundant declarations Eric Wong
@ 2019-05-09 7:51 ` Eric Wong
8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09 7:51 UTC (permalink / raw)
To: meta
This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).
---
MANIFEST | 4 +
ci/README | 33 +++++++
ci/deps.perl | 254 +++++++++++++++++++++++++++++++++++++++++++++++++
ci/profiles.sh | 72 ++++++++++++++
ci/run.sh | 21 ++++
5 files changed, 384 insertions(+)
create mode 100644 ci/README
create mode 100755 ci/deps.perl
create mode 100755 ci/profiles.sh
create mode 100755 ci/run.sh
diff --git a/MANIFEST b/MANIFEST
index da9e364..29cea10 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -27,6 +27,10 @@ MANIFEST
Makefile.PL
README
TODO
+ci/README
+ci/deps.perl
+ci/profiles.sh
+ci/run.sh
contrib/css/216dark.css
contrib/css/216light.css
contrib/css/README
diff --git a/ci/README b/ci/README
new file mode 100644
index 0000000..4687fbc
--- /dev/null
+++ b/ci/README
@@ -0,0 +1,33 @@
+various scripts for automated testing in chroots/VMs/jails
+
+TL;DR: ./ci/run.sh
+
+By default, `sudo' is used to install/uninstall packages. It may be
+overridden with the `SUDO' environment variable. These scripts should
+run in the top-level source tree, that is, as `./ci/run.sh'.
+
+* ci/run.sh - runs tests against all profiles for the current OS
+
+ Environment options may override behavior:
+
+ * DO - may be set to ":" to print commands instead of running
+ default: "" (empty)
+
+ Common commands can be overridden by the environment, too:
+
+ * MAKE - default: "make"
+ * PERL - default: "perl"
+ * SUDO - default: "sudo"
+
+* ci/deps.perl - script to mass-install/remove packages (requires root/sudo)
+
+ Called automatically by ci/run.sh
+
+ There is no need to run this manually unless you are debugging
+ or doing development. However, it can be convenient to for
+ users to mass-install several packages.
+
+* ci/profiles.sh - prints to-be tested package profile for the current OS
+
+ Called automatically by ci/run.sh
+ The output is read by ci/run.sh
diff --git a/ci/deps.perl b/ci/deps.perl
new file mode 100755
index 0000000..e0fda01
--- /dev/null
+++ b/ci/deps.perl
@@ -0,0 +1,254 @@
+#!/usr/bin/perl -w
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# Helper script for installing/uninstalling packages for CI use
+# Intended for use on non-production chroots or VMs since it
+# changes installed packages
+use strict;
+my $usage = "$0 PKG_FMT PROFILE [PROFILE_MOD]";
+my $pkg_fmt = shift;
+@ARGV or die $usage, "\n";
+
+# package profiles
+my $profiles = {
+ # the smallest possible profile
+ # TODO: trim this, Plack pulls in Filesys::Notify::Simple,
+ # and we don't need that for mda-only installs
+ essential => [ qw(
+ git
+ perl
+ Date::Parse
+ Devel::Peek
+ Email::Simple
+ Email::MIME
+ Email::MIME::ContentType
+ Encode
+ Filesys::Notify::Simple
+ Plack
+ URI::Escape
+ ) ],
+
+ # everything optional for normal use
+ optional => [ qw(
+ BSD::Resource
+ DBD::SQLite
+ DBI
+ IO::Compress::Gzip
+ Inline::C
+ Net::Server
+ Plack::Middleware::Deflater
+ Plack::Middleware::ReverseProxy
+ Search::Xapian
+ Socket6
+ highlight.pm
+ xapian-compact
+ ) ],
+
+ # developer stuff
+ devtest => [ qw(
+ IPC::Run
+ Test::HTTP::Server::Simple
+ XML::Feed
+ curl
+ w3m
+ ) ],
+};
+
+# account for granularity differences between package systems and OSes
+my @precious;
+if ($^O eq 'freebsd') {
+ @precious = qw(perl curl Socket6 IO::Compress::Gzip);
+}
+
+if (@precious) {
+ my $re = join('|', map { quotemeta($_) } @precious);
+ for my $list (values %$profiles) {
+ @$list = grep(!/\A(?:$re)\z/, @$list);
+ }
+ push @{$profiles->{essential}}, @precious;
+}
+
+
+# bare minimum for v2
+$profiles->{v2essential} = [ @{$profiles->{essential}}, qw(
+ DBD::SQLite
+ DBI
+ Search::Xapian
+ xapian-compact
+) ];
+
+# package names which can't be mapped automatically:
+my $non_auto = {
+ 'perl' => { pkg => 'perl5' },
+ 'Date::Parse' => {
+ deb => 'libtimedate-perl',
+ pkg => 'p5-TimeDate',
+ rpm => 'perl-TimeDate',
+ },
+ 'Devel::Peek' => {
+ deb => 'perl', # libperl5.XX, but the XX varies
+ pkg => 'perl5',
+ },
+ 'Encode' => {
+ deb => 'perl', # libperl5.XX, but the XX varies
+ pkg => 'perl5',
+ rpm => 'perl-Encode',
+ },
+ 'IO::Compress::Gzip' => {
+ deb => 'perl', # perl-modules-5.xx
+ pkg => 'perl5',
+ rpm => 'perl-PerlIO-gzip',
+ },
+ 'DBD::SQLite' => { deb => 'libdbd-sqlite3-perl' },
+ 'URI::Escape' => {
+ deb => 'liburi-perl',
+ pkg => 'p5-URI',
+ rpm => 'perl-URI',
+ },
+ 'highlight.pm' => {
+ deb => 'libhighlight-perl',
+ pkg => [],
+ rpm => [],
+ },
+
+ # we call xapian-compact(1) in public-inbox-compact(1)
+ 'xapian-compact' => {
+ deb => 'xapian-tools',
+ pkg => 'xapian-core',
+ rpm => 'xapian-core', # ???
+ },
+
+ # OS-specific
+ 'IO::KQueue' => {
+ deb => [],
+ pkg => 'p5-IO-KQueue',
+ rpm => [],
+ },
+};
+
+my (@pkg_install, @pkg_remove, %all);
+for my $ary (values %$profiles) {
+ $all{$_} = \@pkg_remove for @$ary;
+}
+if ($^O eq 'freebsd') {
+ $all{'IO::KQueue'} = \@pkg_remove;
+}
+$profiles->{all} = [ keys %all ]; # pseudo-profile for all packages
+
+# parse the profile list from the command-line
+for my $profile (@ARGV) {
+ if ($profile =~ s/-\z//) {
+ # like apt-get, trailing "-" means remove
+ profile2dst($profile, \@pkg_remove);
+ } else {
+ profile2dst($profile, \@pkg_install);
+ }
+}
+
+# fill in @pkg_install and @pkg_remove:
+while (my ($pkg, $dst_pkg_list) = each %all) {
+ push @$dst_pkg_list, list(pkg2ospkg($pkg, $pkg_fmt));
+}
+
+my @apt_opts =
+ qw(-o APT::Install-Recommends=false -o APT::Install-Suggests=false);
+
+# OS-specific cleanups appreciated
+
+if ($pkg_fmt eq 'deb') {
+ my @quiet = $ENV{V} ? () : ('-q');
+ root('apt-get', @apt_opts, qw(install --purge -y), @quiet,
+ @pkg_install,
+ # apt-get lets you suffix a package with "-" to
+ # remove it in an "install" sub-command:
+ map { "$_-" } @pkg_remove);
+ root('apt-get', @apt_opts, qw(autoremove --purge -y), @quiet);
+} elsif ($pkg_fmt eq 'pkg') {
+ my @quiet = $ENV{V} ? () : ('-q');
+ # FreeBSD, maybe other *BSDs are similar?
+
+ # don't remove stuff that isn't installed:
+ exclude_uninstalled(\@pkg_remove);
+ root(qw(pkg remove -y), @quiet, @pkg_remove) if @pkg_remove;
+ root(qw(pkg install -y), @quiet, @pkg_install) if @pkg_install;
+ root(qw(pkg autoremove -y), @quiet);
+# TODO: yum / rpm support
+} else {
+ die "unsupported package format: $pkg_fmt\n";
+}
+exit 0;
+
+
+# map a generic package name to an OS package name
+sub pkg2ospkg {
+ my ($pkg, $fmt) = @_;
+
+ # check explicit overrides, first:
+ if (my $ospkg = $non_auto->{$pkg}->{$fmt}) {
+ return $ospkg;
+ }
+
+ # check common Perl module name patterns:
+ if ($pkg =~ /::/ || $pkg =~ /\A[A-Z]/) {
+ if ($fmt eq 'deb') {
+ $pkg =~ s/::/-/g;
+ $pkg =~ tr/A-Z/a-z/;
+ return "lib$pkg-perl";
+ } elsif ($fmt eq 'rpm') {
+ $pkg =~ s/::/-/g;
+ return "perl-$pkg"
+ } elsif ($fmt eq 'pkg') {
+ $pkg =~ s/::/-/g;
+ return "p5-$pkg"
+ } else {
+ die "unsupported package format: $fmt for $pkg\n"
+ }
+ }
+
+ # use package name as-is (e.g. 'curl' or 'w3m')
+ $pkg;
+}
+
+# maps a install profile to a package list (@pkg_remove or @pkg_install)
+sub profile2dst {
+ my ($profile, $dst_pkg_list) = @_;
+ if (my $pkg_list = $profiles->{$profile}) {
+ $all{$_} = $dst_pkg_list for @$pkg_list;
+ } elsif ($all{$profile}) { # $profile is just a package name
+ $all{$profile} = $dst_pkg_list;
+ } else {
+ die "unrecognized profile or package: $profile\n";
+ }
+}
+
+sub exclude_uninstalled {
+ my ($list) = @_;
+ my %inst_check = (
+ pkg => sub { system(qw(pkg info -q), $_[0]) == 0 },
+ deb => sub { system("dpkg -s $_[0] >/dev/null 2>&1") == 0 },
+ rpm => sub { system("rpm -qs $_[0] >/dev/null 2>&1") == 0 },
+ );
+
+ my $cb = $inst_check{$pkg_fmt} || die <<"";
+don't know how to check install status for $pkg_fmt
+
+ my @tmp;
+ for my $pkg (@$list) {
+ push @tmp, $pkg if $cb->($pkg);
+ }
+ @$list = @tmp;
+}
+
+sub root {
+ print join(' ', @_), "\n";
+ return if $ENV{DRY_RUN};
+ return if system(@_) == 0;
+ warn 'command failed: ', join(' ', @_), "\n";
+ exit($? >> 8);
+}
+
+# ensure result can be pushed into an array:
+sub list {
+ my ($pkg) = @_;
+ ref($pkg) eq 'ARRAY' ? @$pkg : $pkg;
+}
diff --git a/ci/profiles.sh b/ci/profiles.sh
new file mode 100755
index 0000000..31140f3
--- /dev/null
+++ b/ci/profiles.sh
@@ -0,0 +1,72 @@
+#!/bin/sh
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Prints OS-specific package profiles to stdout (one per-newline) to use
+# as command-line args for ci/deps.perl. Called automatically by ci/run.sh
+
+# set by os-release(5) or similar
+ID= VERSION_ID=
+case $(uname -o) in
+GNU/Linux)
+ for f in /etc/os-release /usr/lib/os-release
+ do
+ test -f $f || continue
+ . $f
+
+ # Debian sid (and testing) have no VERSION_ID
+ case $ID--$VERSION_ID in
+ debian--)
+ case $PRETTY_NAME in
+ */sid) VERSION_ID=sid ;;
+ *)
+ echo >&2 "$ID, but no VERSION_ID"
+ echo >&2 "==> $f <=="
+ cat >&2 $f
+ exit 1
+ ;;
+ esac
+ ;;
+ esac
+
+ case $ID--$VERSION_ID in
+ -|*--|--*) continue ;;
+ *--*) break ;;
+ esac
+ done
+ ;;
+FreeBSD)
+ ID=freebsd
+ VERSION_ID=$(uname -r | cut -d . -f 1)
+ test "$VERSION_ID" -lt 11 && {
+ echo >&2 "ID=$ID $(uname -r) too old to support";
+ exit 1
+ }
+esac
+
+case $ID in
+freebsd) PKG_FMT=pkg ;;
+debian|ubuntu) PKG_FMT=deb ;;
+centos|redhat|fedora) PKG_FMT=rpm ;;
+*) echo >&2 "PKG_FMT undefined for ID=$ID in $0"
+esac
+
+case $ID-$VERSION_ID in
+freebsd-11|freebsd-12) sed "s/^/$PKG_FMT /" <<EOF
+all devtest-
+all devtest IO::KQueue-
+all devtest IO::KQueue
+v2essential
+essential
+essential devtest-
+EOF
+ ;;
+debian-sid|debian-9|debian-10) sed "s/^/$PKG_FMT /" <<EOF
+all devtest-
+all devtest
+v2essential
+essential
+essential devtest-
+EOF
+ ;;
+esac
diff --git a/ci/run.sh b/ci/run.sh
new file mode 100755
index 0000000..87d9561
--- /dev/null
+++ b/ci/run.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+set -e
+SUDO=${SUDO-'sudo'} PERL=${PERL-'perl'} MAKE=${MAKE-'make'}
+DO=${DO-''}
+
+set -x
+if test -f Makefile
+then
+ $DO $MAKE clean
+fi
+
+./ci/profiles.sh | while read args
+do
+ $DO $SUDO $PERL -w ci/deps.perl $args
+ $DO $PERL Makefile.PL
+ $DO $MAKE
+ $DO $MAKE check
+ $DO $MAKE clean
+done
--
EW
^ permalink raw reply related [flat|nested] 10+ messages in thread