interop with leafnode

interop with leafnode

[Dave Taht]

Jon Corbet turned me onto his nntp server instance which archives a
bunch of mailing lists and leverages public-inbox.

I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
Blazing fast, useful at a glance.

In a paroxysm of overenthusiasm for the the good ole days I then setup a
leafnode real quick and pointed it at his instance so I could replicate...

For most (not all) traffic, leafnode does this:

> Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> considering articles 200621 - 200643
> Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> - no Path:  found

:(

I'm really glad to hear of this project. It sounds like more of the
right way to do this is via a local git?

Re: interop with leafnode

[Eric Wong]

Dave Taht <dave@taht.net> wrote:
> I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
> Blazing fast, useful at a glance.
> 
> In a paroxysm of overenthusiasm for the the good ole days I then setup a
> leafnode real quick and pointed it at his instance so I could replicate...
> 
> For most (not all) traffic, leafnode does this:
> 
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> > considering articles 200621 - 200643
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> > - no Path:  found

I guess there's some bug or missing commands in public-inbox-nntpd
which leafnode relies on.  Sorry I haven't had time to investigate
or do much with public-inbox, lately (but maybe you could :)

> :(
> 
> I'm really glad to hear of this project. It sounds like more of the
> right way to do this is via a local git?

Yes, git clone/fetch + running your own public-inbox-nntpd instance
and using slrn/gnus/mutt(w/ nntp patch)/sylpheed/thunderbird all
seem to work.

Re: interop with leafnode

[Dave Taht]

Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
>> Blazing fast, useful at a glance.
>> 
>> In a paroxysm of overenthusiasm for the the good ole days I then
>> setup a
>> leafnode real quick and pointed it at his instance so I could
>> replicate...
>> 
>> For most (not all) traffic, leafnode does this:
>> 
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
>> > considering articles 200621 - 200643
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article
>> > 200621
>> > - no Path:  found
>
> I guess there's some bug or missing commands in public-inbox-nntpd
> which leafnode relies on.  Sorry I haven't had time to investigate
> or do much with public-inbox, lately (but maybe you could :)
>
>> :(
>> 
>> I'm really glad to hear of this project. It sounds like more of the
>> right way to do this is via a local git?
>
> Yes, git clone/fetch + running your own public-inbox-nntpd instance
> and using slrn/gnus/mutt(w/ nntp patch)/sylpheed/thunderbird all
> seem to work.

I will try to modernize. 

Re: interop with leafnode

[Eric Wong]

Dave Taht <dave@taht.net> wrote:
> I will try to modernize. 

It's probably only missing a few commands to support leafnode
or other stuff (e.g. NNTP replication with innd).  I just don't
have experience with leafnode, and lately I've been fighting
with hardware problems, bills and taxes :<

Re: interop with leafnode

[Dave Taht]

Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> I will try to modernize. 
>
> It's probably only missing a few commands to support leafnode
> or other stuff (e.g. NNTP replication with innd).  I just don't
> have experience with leafnode, and lately I've been fighting
> with hardware problems, bills and taxes :<

Back in the day I used to maintain an innd instance, too.

I am tempted to start up a news instance again, just because
the traffic rate is now so low and because I miss things like comp.arch

Re: interop with leafnode

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> I guess there's some bug or missing commands in public-inbox-nntpd
> which leafnode relies on.  Sorry I haven't had time to investigate
> or do much with public-inbox, lately (but maybe you could :)

I still haven't tried leafnode, but I just remembered I fixed an
NNTP bug a while back to give it compatibility with Alpine:

https://public-inbox.org/meta/20190116045802.ikm67b4cwigso3tc@dcvr/

commit d66aa534a4a7506cfc5cfab49d1e09f8db8be3dd
("nntp: header responses use CRLF consistently")

Perhaps it's the same bug you hit with leafnode?

Re: interop with leafnode

[Eric Wong]

Dave Taht <dave@taht.net> wrote:
> For most (not all) traffic, leafnode does this:
> 
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> > considering articles 200621 - 200643
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> > - no Path:  found

So yeah, I Cc-ed you on <20190613065413.4705-1-e@80x24.org>.
Apparently leafnode doesn't need any real Path:, and just a
single byte seems to work.

However, I've also noticed leafnode (finally tried it) also
fails if a Message-ID is too long and spans multiple lines
(those were from outlook.com, go figure).

Re: interop with leafnode

[Dave Taht]

Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> For most (not all) traffic, leafnode does this:
>> 
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
>> > considering articles 200621 - 200643
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article
>> > 200621
>> > - no Path:  found
>
> So yeah, I Cc-ed you on <20190613065413.4705-1-e@80x24.org>.
> Apparently leafnode doesn't need any real Path:, and just a
> single byte seems to work.
>
> However, I've also noticed leafnode (finally tried it) also
> fails if a Message-ID is too long and spans multiple lines
> (those were from outlook.com, go figure).

Thanks! Is there a way to reformat that?

[PATCH] nntp: ensure Message-ID is not folded for leafnode

[Eric Wong]

Dave Taht <dave@taht.net> wrote:
> Eric Wong <e@80x24.org> writes:
> > However, I've also noticed leafnode (finally tried it) also
> > fails if a Message-ID is too long and spans multiple lines
> > (those were from outlook.com, go figure).
> 
> Thanks! Is there a way to reformat that?

Yeah, this seems to work:

-----------8<------------
Subject: [PATCH] nntp: ensure Message-ID is not folded for leafnode

Leafnode cannot handle Message-ID headers which are too long and
require folding via Email::Simple::Header.  Since there are
already many of these messages in git with the header already
folded, we need to handle the unfolding when emitting the
message via NNTP.

As far as we know, Leafnode is the only client software
incapable of handling this case.
---
 lib/PublicInbox/NNTP.pm |  7 +++++++
 t/nntpd.t               | 20 ++++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 1d07220..d409e78 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -519,6 +519,13 @@ sub _header ($) {
 	my $hdr = $_[0]->header_obj->as_string;
 	utf8::encode($hdr);
 	$hdr =~ s/(?<!\r)\n/\r\n/sg;
+
+	# for leafnode compatibility, we need to ensure Message-ID headers
+	# are only a single line.  We can't subclass Email::Simple::Header
+	# and override _default_fold_at in here, either; since that won't
+	# affect messages already in the archive.
+	$hdr =~ s/^(Message-ID:)[ \t]*\r\n[ \t]+([^\r]+)\r\n/$1 $2\r\n/igsm;
+
 	$hdr
 }
 
diff --git a/t/nntpd.t b/t/nntpd.t
index c73cc12..cce21ee 100644
--- a/t/nntpd.t
+++ b/t/nntpd.t
@@ -231,6 +231,26 @@ EOF
 		ok($date >= $t0, 'valid date after start');
 		ok($date <= $t1, 'valid date before stop');
 	}
+	if ('leafnode interop') {
+		my $for_leafnode = PublicInbox::MIME->new(<<"");
+From: longheader\@example.com
+To: $addr
+Subject: none
+Date: Fri, 02 Oct 1993 00:00:00 +0000
+
+		my $long_hdr = 'for-leafnode-'.('y'x200).'@example.com';
+		$for_leafnode->header_set('Message-ID', "<$long_hdr>");
+		$im->add($for_leafnode);
+		$im->done;
+		if ($version == 1) {
+			my $s = PublicInbox::SearchIdx->new($mainrepo, 1);
+			$s->index_sync;
+		}
+		my $hdr = $n->head("<$long_hdr>");
+		my $expect = qr/\AMessage-ID: /i . qr/\Q<$long_hdr>\E/;
+		ok(scalar(grep(/$expect/, @$hdr)), 'Message-ID not folded');
+		ok(scalar(grep(/^Path:/, @$hdr)), 'Path: header found');
+	}
 
 	# pipelined requests:
 	{
-- 
EW