From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id CB2MOlUFsmFW7gAAgWs5BA (envelope-from ) for ; Thu, 09 Dec 2021 14:32:05 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id WGBmNlUFsmEaKwAA1q6Kng (envelope-from ) for ; Thu, 09 Dec 2021 13:32:05 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B89413376F for ; Thu, 9 Dec 2021 14:32:05 +0100 (CET) Received: from localhost ([::1]:39592 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mvJWm-00037k-TG for larch@yhetil.org; Thu, 09 Dec 2021 08:32:04 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34962) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvJSs-0000ud-PD for bug-guix@gnu.org; Thu, 09 Dec 2021 08:28:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59560) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mvJSs-00088p-Hl for bug-guix@gnu.org; Thu, 09 Dec 2021 08:28:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mvJSs-0002ot-Ew for bug-guix@gnu.org; Thu, 09 Dec 2021 08:28:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#52338: Crawler bots are downloading substitutes Resent-From: Mathieu Othacehe Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 09 Dec 2021 13:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52338 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Leo Famulari Received: via spool by 52338-submit@debbugs.gnu.org id=B52338.163905646910819 (code B ref 52338); Thu, 09 Dec 2021 13:28:02 +0000 Received: (at 52338) by debbugs.gnu.org; 9 Dec 2021 13:27:49 +0000 Received: from localhost ([127.0.0.1]:42873 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mvJSe-0002oR-QZ for submit@debbugs.gnu.org; Thu, 09 Dec 2021 08:27:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34132) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mvJSd-0002oF-9L for 52338@debbugs.gnu.org; Thu, 09 Dec 2021 08:27:47 -0500 Received: from [2001:470:142:3::e] (port=42764 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvJSX-00086l-Vi; Thu, 09 Dec 2021 08:27:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=AIpbgcpe3biXFQ7sZcsjdHlc7o2rRWDgsqdPmGA8LTE=; b=b8MWdka0M9rIWS3riZBn DwFWrwJchGU9IuCmuFeGY3PwXBcUUex+kwLZKYU0OC9sfE3we5JigRIZ07o1B2Kz9WLOm45g9OR1F O8gy6ZkPTwuY6xsQStulztUeOHxthKEun2B6yHMZPyPvv2s3sCr/36kOOHqPhXsoVpPXeiqRQOzM/ RPG+g3zxIu/KheshgjEDUjMvOjX944XDnQv2GOVfG58dcNtK2TKyyoeEhxiNSMNpST7E0uLtX+LVd egbMDSunsW2FoIsX509/SPDmqlfTE84QU4/UVVbE+dVKTWrVY/Z60QiS3vPyS7gL8D0XP7GQd34St K5ms8vWCDBUYZQ==; Received: from lfbn-lyo-1-2042-171.w90-66.abo.wanadoo.fr ([90.66.207.171]:49532 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvJSX-0000z8-0A; Thu, 09 Dec 2021 08:27:41 -0500 From: Mathieu Othacehe References: <2f52f6b48db55f8a79b07dbb242b297ab49d6083.1638828946.git.leo@famulari.name> Date: Thu, 09 Dec 2021 14:27:38 +0100 In-Reply-To: <2f52f6b48db55f8a79b07dbb242b297ab49d6083.1638828946.git.leo@famulari.name> (Leo Famulari's message of "Mon, 6 Dec 2021 17:18:10 -0500") Message-ID: <87tufh6h85.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 52338@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639056725; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=AIpbgcpe3biXFQ7sZcsjdHlc7o2rRWDgsqdPmGA8LTE=; b=RR3XNN1DfkQQ/5NguzZqnAF3xIopGCldeAYhfNCB5J0TOP51qI35R9An50pw/RhIpN6Lmq sriihV7GdCs/UyQiucBmsApunnvX3R5fesLpdjRqVcmOfMA/cie02a9zX6CF6zrl1ElB/C xVWZ5B75yrEQnoUBUNc2+2WG5r+cWw3IgLNwv2AYRoNI5FwOF2bLK5e+gw4M2xB3W+zekb C2BHS7qv1mLM6iSuBlMbVOzrOPpsEXSBfd0oeWsuSMLhL7D38xVwVODq1yYuGxFI7XYBSR cVHobb/GMwyi39IFKRALohgupHWiPq4g66OnOYzlh59Hkp5bu6TXFMJU+yFKkQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639056725; a=rsa-sha256; cv=none; b=UB49Ak2xKB3mJ7aeUcGXVz4zxvSHyRY9aIeugySQy+HeE7r5r3tLzyxGoOrHyUSZ1N1AvQ C1QGa+C7JaNU28uenaEmFRGs/M7CxFITKDnEYRebt6SxhFHh0B1vHfI0ZRdEkUBIxNdf5Z VQa/lqZyXENlMoBQshgP6fQ9AoG4jiN2mpcAZOb4TsCgz6TgMJ8HIOGj3OHwo2PljZJdS9 om4I9isUbDWVESA3hZ6PtGn8J2gw/pBtUxcnSGhFjnXcIWl2BNaMOV+VGJD7SzNRltaebu sSlCcv5oGNd8KywDAVmV1WensG7bzIJsYFRY0M90Rj+r8WXbA4TEgOxHULhcPQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=b8MWdka0; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.65 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=b8MWdka0; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: B89413376F X-Spam-Score: -3.65 X-Migadu-Scanner: scn1.migadu.com X-TUID: m+aNYNlurVC1 Hello Leo, > + (nginx-location-configuration > + (uri "/robots.txt") > + (body > + (list > + "add_header Content-Type text/plain;" > + "return 200 \"User-agent: *\nDisallow: /nar/\n\";")))))) Nice, the bots are also accessing the Cuirass web interface, do you think it would be possible to extend this snippet to prevent it? Thanks, Mathieu