From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <kyle@kyleam.com>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
	SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham
	autolearn_force=no version=3.4.2
Received: from out2.migadu.com (out2.migadu.com [IPv6:2001:41d0:2:aacc::])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by dcvr.yhbt.net (Postfix) with ESMTPS id 6384E1F934
	for <meta@public-inbox.org>; Sun, 24 Jan 2021 06:01:12 +0000 (UTC)
X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers.
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kyleam.com; s=key1;
	t=1611468068;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=kWOMQQscrRlfHK4lMLS8LHmsJuoT2nmQVZzbCIDXqMw=;
	b=AekoE4+1XKocKPHuHOTpzpZPHCKNCCN7Nthg8dOnYLCbgRfDcSM04EuaB9TiPoFzV5ruiL
	21sqAX2ZTyYYG9F6LTTHTIX2j+wiZ5CeFq4SdkvKQUngum+47djolT1DJGaJJY/lAV3JsI
	1VfQV0QJQHA+LCr4iDaRr7C8iG3j8e4PfqgwlLiLpSBhu2MQsqs6x0oTyvwC5BQBnZz2b/
	nQMp7NR0UNveYsyfxlcfWMROGFyvLS0j88BU06xa9WdrmcENIBVvMnrIGileU5w83ru2wM
	EoXX56D3Z/RpOjrFZAOoEENbD+zDzNOLF5lIJTz94DprYVrR4vam/9W9NQSBZg==
From: Kyle Meyer <kyle@kyleam.com>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH 02/10] lei: support remote externals
In-Reply-To: <20210123102755.425-3-e@80x24.org>
References: <20210123102755.425-1-e@80x24.org>
 <20210123102755.425-3-e@80x24.org>
Date: Sun, 24 Jan 2021 01:01:02 -0500
Message-ID: <87v9bmswkh.fsf@kyleam.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Migadu-Flow: FLOW_OUT
X-Migadu-Auth-User: kyle@kyleam.com
List-Id: <meta.public-inbox.org>

Eric Wong writes:

>  sub query_remote_mboxrd {
[...]
> +	my @cmd = qw(curl -XPOST -sSf);

I've been playing around with lei locally (wow :>).

The one snag I've hit is hooking up the http archives that I host
(<https://yhetil.org>).  It seems to boil down to the `curl -XPOST'
command failing.  For example, this works fine with public-inbox.org:

  $ curl -sSf -XPOST 'https://public-inbox.org/meta/?q=s:lei&x=m' | zless

But it fails with the mirror of meta at yhetil.org:

  $ curl -sSf -XPOST 'https://yhetil.org/meta/?q=s:lei&x=m' | zless
  curl: (22) The requested URL returned error: 400 Bad Request

If I add -d'' to the call, it works and produces the same output as the
above call against public-inbox.org/meta.  And if I add this option to
query_remote_mboxrd (i.e. applying the change at the end), `lei q' works
for me as expected.

yhetil.org uses nginx and varnish, and I'm _very_ far from being an
expert in either of those, so I have no doubt that the above error could
be the result of me configuring something incorrectly.  However, despite
a fair amount of time and effort, I couldn't figure out how to tweak
things to make the above command work without --data.

I quickly checked <https://lore.kernel.org>, and it seems like it would
give a similar response to -XPOST without data:

  $ curl -fSs -XPOST 'https://lore.kernel.org/git/?q=get-urlmatch&x=m'
  curl: (22) The requested URL returned error: 400
  $  curl -d'' -fSs -XPOST 'https://lore.kernel.org/git/?q=get-urlmatch&x=m' | zless
  # works


diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index defe5e67..766e9f5f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -192,7 +192,7 @@ sub query_remote_mboxrd {
 	my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $uri);
 	my $dedupe = $lei->{dedupe} // die 'BUG: {dedupe} missing';
 	$dedupe->prepare_dedupe;
-	my @cmd = qw(curl -XPOST -sSf);
+	my @cmd = qw(curl -XPOST -d'' -sSf);
 	$opt->{torsocks} = 'false' if $opt->{'no-torsocks'};
 	my $tor = $opt->{torsocks} //= 'auto';
 	if ($tor eq 'auto' && substr($uri->host, -6) eq '.onion' &&