From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id CFmIOQR/rmEiggAAgWs5BA (envelope-from ) for ; Mon, 06 Dec 2021 22:22:12 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 8P0dNQR/rmHoEgAAbx9fmQ (envelope-from ) for ; Mon, 06 Dec 2021 21:22:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AEBF49D6E for ; Mon, 6 Dec 2021 22:22:12 +0100 (CET) Received: from localhost ([::1]:51772 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1muLR5-0002fj-Rn for larch@yhetil.org; Mon, 06 Dec 2021 16:22:11 -0500 Received: from eggs.gnu.org ([209.51.188.92]:50246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muLQw-0002fM-Aw for bug-guix@gnu.org; Mon, 06 Dec 2021 16:22:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:52255) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1muLQw-0001cp-3J for bug-guix@gnu.org; Mon, 06 Dec 2021 16:22:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1muLQv-00015J-TP for bug-guix@gnu.org; Mon, 06 Dec 2021 16:22:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#52338: Crawler bots are downloading substitutes Resent-From: Leo Famulari Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 06 Dec 2021 21:22:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 52338 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 52338@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16388256634100 (code B ref -1); Mon, 06 Dec 2021 21:22:01 +0000 Received: (at submit) by debbugs.gnu.org; 6 Dec 2021 21:21:03 +0000 Received: from localhost ([127.0.0.1]:35568 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1muLPy-000144-Pb for submit@debbugs.gnu.org; Mon, 06 Dec 2021 16:21:02 -0500 Received: from lists.gnu.org ([209.51.188.17]:53410) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1muLPx-00013p-R3 for submit@debbugs.gnu.org; Mon, 06 Dec 2021 16:21:02 -0500 Received: from eggs.gnu.org ([209.51.188.92]:50116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muLPx-0002b6-JL for bug-guix@gnu.org; Mon, 06 Dec 2021 16:21:01 -0500 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:36171) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muLPv-0001Ru-Sv for bug-guix@gnu.org; Mon, 06 Dec 2021 16:21:01 -0500 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 87E375C0233; Mon, 6 Dec 2021 16:20:58 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 06 Dec 2021 16:20:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:subject:message-id:mime-version:content-type; s= mesmtp; bh=i0j0URj5r82LVUNhFfK90P6QvKS+AA4MQCr5LyiR/M4=; b=shmVI OgNOyHu0tnwpFkKCbeESUsMUVnEZX/+8Od/WLo44ZGtTROtv/RXZDRJZJC+zRd0n 02OP/uqPlZjc5i+XaQYIxorGQbtOGjuFZRwnoSy7OATDpbGX9Vaae27PlgryehdF HqiI0lMR39mhElAqrA26BDUV3pXDjJij3/tCkE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=i0j0URj5r82LVUNhFfK90P6QvKS+A A4MQCr5LyiR/M4=; b=hUwiaYFogsArb++AQdfOjqQY0SUkSsbJUPC+lRnaaeeTY XOqb1QsL/ISso31u6XlhJ09zdPTLThIr+CkpyK5mndWkWDQSRh0v+3L7/ew8wQf4 ROQLIfEJtffrzz3dRwboAAi2L4q/Vgvc9qS8niyn/IW0FtqyCnI3mYeJTPfR5ePi ONGmoh1KuMKpDW9O1Y2Bvm10rAnzj7vUk6wsnntK5CDhWvO3/llM3aCa9JGRrkF5 guWVzFMIv0uA5V6AD+0on/F6MxZ6sVf7zv/8INkQvsy0ikG2FhK8X9Fd0dIkemEa WT9ffZPK4opVGSf1Rp/lnMQBLXiGXexkoromLC5wQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrjeefgddugeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesthdtredttd dtvdenucfhrhhomhepnfgvohcuhfgrmhhulhgrrhhiuceolhgvohesfhgrmhhulhgrrhhi rdhnrghmvgeqnecuggftrfgrthhtvghrnhepfeeuueetgfdtleehieeugeetieekhffhge ekheekgfdtkeekkefhtdegleekvedvnecuffhomhgrihhnpehgnhhurdhorhhgpdgsihhn ghdrtghomhdpshgvmhhruhhshhdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehlvghosehfrghmuhhlrghrihdrnhgrmhgv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 6 Dec 2021 16:20:58 -0500 (EST) Date: Mon, 6 Dec 2021 16:20:55 -0500 From: Leo Famulari Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Received-SPF: pass client-ip=66.111.4.26; envelope-from=leo@famulari.name; helo=out2-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1638825732; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=i0j0URj5r82LVUNhFfK90P6QvKS+AA4MQCr5LyiR/M4=; b=iM/AdMsw3lFHkDGq6qj6aA0VzC4ag/083Iu+z2u+vD5mnhtb3bouNy+2AXiqvjgLabRoTq 9vDgP6zl2+hlxRFrXDQ3wPrFMGH20ef86GtW6az9U8XddVLU3ejubpOefguVOJUX34UeiL MtLOKmmNLDuk27sJM4UJlPfpyS+S7ytla1ipGkZ6ZDhaZ67xb81ywnyasA7syRpqjEDDTi WpP9XmXtFGkuKAIbqy3W6lnbMOKd7iLX8Jjgh+9sADx/KCJ4GgDRrQdVGifEuTRz9ELbc+ EjE7obNKEJAn3AgoC4P61lsfGCtW9r6hfzn0Gp9mZsP5OZsUyLeml3DfjXrhaA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638825732; a=rsa-sha256; cv=none; b=sL+RYaUoscVQboGksNuLgUr1H2iD8h71hlpraa1sS01tB3TEq+D9n0N9zTe2dmbbYgwl9t 28Qkz8ZuLi0qgCwackUnmbDPVmWpRRd8rpzr4Z7TW85X0fg1QvZpl+s3AGwQs0/JdLQ5Kb EQDd4TLWNoYs+P/VJu5i38ZF4b+xa1vCug8xdKOHtZgZL/BK0I8xxlCy+eHUn7l5ycOR94 4kM8GjknWdoeD8HFjt5W819Sjv4kRWrSCPTsbEUNwPWISqRSgN6hq/j7qht0H1lIIONQIv MOJc6ItxD3Eb9dLnJnJDa8vqz+++Nt+y3PNxepHMxZsw75kjf8CjtnracrWVZg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=famulari.name header.s=mesmtp header.b="shmVI Og"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=hUwiaYFo; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.24 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=famulari.name header.s=mesmtp header.b="shmVI Og"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=hUwiaYFo; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: AEBF49D6E X-Spam-Score: -2.24 X-Migadu-Scanner: scn0.migadu.com X-TUID: ldrqu8MRHJMF I noticed that some bots are downloading substitutes from ci.guix.gnu.org. We should add a robots.txt file to reduce this waste. Specifically, I see bots from Bing and Semrush: https://www.bing.com/bingbot.htm https://www.semrush.com/bot.html