* [ANNOUNCE] public-inbox 1.6.0
@ 2020-09-16 20:03 Eric Wong
2020-09-19 20:01 ` Leah Neukirchen
0 siblings, 1 reply; 4+ messages in thread
From: Eric Wong @ 2020-09-16 20:03 UTC (permalink / raw)
To: meta
A big release containing several performance optimizations, a
new anonymous IMAP server, and more. It represents an
incremental improvement over 1.5 in several areas with more to
come in 1.7.
The read-only httpd and nntpd daemons no longer block the event
loop when retrieving blobs from git, making better use of SMP
systems while accomodating slow storage.
Indexing can be now be tuned to give somewhat usable performance
on HDD storage, though we can't defy the laws of physics, either.
* General changes:
- ~/.cache/public-inbox/inline-c is automatically used for Inline::C
if it exists. PERL_INLINE_DIRECTORY in env remains supported
and prioritized to support `nobody'-type users without HOME.
- msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3
behavior for a minor reduction in VFS traffic
- public-inbox-tuning(7) - new manpage containing pointers to
various tuning options and tips for certain HW and OS setups.
- Copy-on-write is disabled on BTRFS for new indices to avoid
fragmentation. See the new public-inbox-tuning(7) manpage.
- message/{rfc822,news,global} attachments are decoded recursively
and indexed for search. Reindexing (see below) is required
to ensure these attachments are indexed in old messages.
- inbox.lock (v2) and ssoma.lock (v1) files are written to
on message delivery (or spam removal) to wake up read-only
daemons via inotify or kqueue.
- `--help' switch supported by command-line tools
* Upgrading for new features in 1.6
The ordering of these steps is only necessary if you intend to
use some new features in 1.6. Future releases may have
different instructions (or be entirely transparent).
0. install (use your OS package manager, or "make install")
1. restart public-inbox-watch instances if you have any
2. Optional: remove Plack::Middleware::Deflater if you're using
a custom .psgi file for PublicInbox::WWW. This only saves
some memory and CPU cycles, and you may also skip this step
if you expect to roll back to 1.5.0 for any reason.
Steps 3a and 3b may happen in any order, 3b is optional
and is only required to use new WWW and IMAP features.
3a. restart existing read-only daemons if you have them
(public-inbox-nntpd, public-inbox-httpd)
3b. run "public-inbox-index -c --reindex --rethread --all"
to reindex all configured inboxes
4. configure and start the new public-inbox-imapd. This
requires reindexing in 3b, but there's no obligation to
run an IMAP server, either.
* public-inbox-index
There are several new options to improve usability on slow,
rotational storage.
- `--batch-size=BYTES' or publicinbox.indexBatchSize parameter
to reduce frequency of random writes on HDDs
- `--sequential-shard' or publicInbox.sequentialShard parameter
to improve OS page cache utilization on HDDs.
- `--no-fsync' when combined with Xapian 1.4+ can be used to
speed up indexing on SSDs and small (default) `--batch-size'
- `--rethread' option to go with `--reindex' (use sparringly,
see manpage)
- parallelize v2 updates by default, `--sequential-shard' and
`-j0' is (once again) allowed to disable parallelization
- (re-)indexing parallelizes blob reads from git
- `--all' may be specified to index all configured inboxes
* public-inbox-learn
- `rm' supports `--all' to remove from all configured inboxes
* public-inbox-imapd
- new read-only IMAP daemon similar to public-inbox-nntpd
`AUTH=ANONYMOUS' is supported, but any username and
password for clients without `AUTH=ANONYMOUS' support.
* public-inbox-nntpd
- blob reads from git are handled asynchronously
* public-inbox-httpd
- Plack::Middleware::Deflater is no longer loaded by default
when no .psgi file is specified; PublicInbox::WWW can rely
on gzip for buffering (see below)
* PublicInbox::WWW
- use consistent blank line around attachment links
- Attachments in message/{rfc822,news,global} messages can be
individually downloaded. Downloading the entire message/rfc822
file in full remains supported
- $INBOX_DIR/description is treated as UTF-8
- HTML, Atom, and text/plain responses are gzipped without
relying on Plack::Middleware::Deflater
- Multi-message endpoints (/t.mbox.gz, /T/, /t/, etc) are ~10% faster
when running under public-inbox-httpd with asynchronous blob
retrieval
- mbox search results may now include all messages pertaining to that
thread. Needs `--reindex' mentioned above in
`Upgrading for new features in 1.6'.
- fix mbox.gz search results downloads for lynx users
- small navigation tweaks, more prominent mirroring instructions
* public-inbox-watch
- Linux::Inotify2 or IO::KQueue is used directly,
Filesys::Notify::Simple is no longer required
- NNTP groups and IMAP mailboxes may be watched in addition
to Maildirs (lightly tested).
* Ongoing internal changes
- reduce event loop hogging for many-inbox support
- use more Perl v5.10-isms, future-proof against Perl 8
- more consistent variable and field naming, improve internal
documentation and comments
- start supporting >=40 char git identifiers for SHA-256
- test -httpd-specific code paths via Plack::Test::ExternalServer
in addition to generic PSGI paths.
Please report bugs via plain-text mail to: meta@public-inbox.org
See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ANNOUNCE] public-inbox 1.6.0
2020-09-16 20:03 [ANNOUNCE] public-inbox 1.6.0 Eric Wong
@ 2020-09-19 20:01 ` Leah Neukirchen
2020-09-19 21:17 ` Eric Wong
0 siblings, 1 reply; 4+ messages in thread
From: Leah Neukirchen @ 2020-09-19 20:01 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Hi,
thanks for the release!
Eric Wong <e@80x24.org> writes:
> * Upgrading for new features in 1.6
>
> The ordering of these steps is only necessary if you intend to
> use some new features in 1.6. Future releases may have
> different instructions (or be entirely transparent).
>
> 0. install (use your OS package manager, or "make install")
>
> 1. restart public-inbox-watch instances if you have any
>
> 2. Optional: remove Plack::Middleware::Deflater if you're using
> a custom .psgi file for PublicInbox::WWW. This only saves
> some memory and CPU cycles, and you may also skip this step
> if you expect to roll back to 1.5.0 for any reason.
>
> Steps 3a and 3b may happen in any order, 3b is optional
> and is only required to use new WWW and IMAP features.
>
> 3a. restart existing read-only daemons if you have them
> (public-inbox-nntpd, public-inbox-httpd)
>
> 3b. run "public-inbox-index -c --reindex --rethread --all"
> to reindex all configured inboxes
>
> 4. configure and start the new public-inbox-imapd. This
> requires reindexing in 3b, but there's no obligation to
> run an IMAP server, either.
I did all these steps in this order, NNTP works fine but IMAP shows
all folders as empty. Any ideas how to debug this?
thx,
--
Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ANNOUNCE] public-inbox 1.6.0
2020-09-19 20:01 ` Leah Neukirchen
@ 2020-09-19 21:17 ` Eric Wong
2020-09-19 21:24 ` Leah Neukirchen
0 siblings, 1 reply; 4+ messages in thread
From: Eric Wong @ 2020-09-19 21:17 UTC (permalink / raw)
To: Leah Neukirchen; +Cc: meta
Leah Neukirchen <leah@vuxu.org> wrote:
> Hi,
>
> thanks for the release!
You're welcome!
> > * Upgrading for new features in 1.6
<snip>
> I did all these steps in this order, NNTP works fine but IMAP shows
> all folders as empty. Any ideas how to debug this?
Any chance you're hitting "$NEWSGROUP" and not "$NEWSGROUP.0"?
(or ".1", ".2" ...)?
I had to split the mailboxes into slices (".0"-".$N") to deal
with client-side limitations; so "$NEWSGROUP" is just an empty
folder with sub-folders containing messages.
Otherwise, stdout should have a trace of commands issued by the
client, and strace/truss/tcpdump/etc should show more...
It's been a while since I've really touched this part of the
IMAP code (and it's been a long year :x), but I don't think I
ever saw something like this while working on it.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [ANNOUNCE] public-inbox 1.6.0
2020-09-19 21:17 ` Eric Wong
@ 2020-09-19 21:24 ` Leah Neukirchen
0 siblings, 0 replies; 4+ messages in thread
From: Leah Neukirchen @ 2020-09-19 21:24 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong <e@80x24.org> writes:
> Leah Neukirchen <leah@vuxu.org> wrote:
>> Hi,
>>
>> thanks for the release!
>
> You're welcome!
>
>> > * Upgrading for new features in 1.6
>
> <snip>
>
>> I did all these steps in this order, NNTP works fine but IMAP shows
>> all folders as empty. Any ideas how to debug this?
>
> Any chance you're hitting "$NEWSGROUP" and not "$NEWSGROUP.0"?
> (or ".1", ".2" ...)?
>
> I had to split the mailboxes into slices (".0"-".$N") to deal
> with client-side limitations; so "$NEWSGROUP" is just an empty
> folder with sub-folders containing messages.
Of course, that was it. Works fine now. :)
--
Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-09-19 21:24 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-16 20:03 [ANNOUNCE] public-inbox 1.6.0 Eric Wong
2020-09-19 20:01 ` Leah Neukirchen
2020-09-19 21:17 ` Eric Wong
2020-09-19 21:24 ` Leah Neukirchen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).