all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix@gnu.org>
To: Mark H Weaver <mhw@netris.org>
Cc: 52338@debbugs.gnu.org, leo@famulari.name
Subject: bug#52338: Crawler bots are downloading substitutes
Date: Fri, 10 Dec 2021 23:52:51 +0100	[thread overview]
Message-ID: <875yrw3vvk.fsf@nckx> (raw)
In-Reply-To: <87r1ak2m1p.fsf@netris.org>

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

All,

Mark H Weaver 写道:
> For what it's worth: during the years that I administered Hydra, 
> I found
> that many bots disregarded the robots.txt file that was in place 
> there.
> In practice, I found that I needed to periodically scan the 
> access logs
> for bots and forcefully block their requests in order to keep 
> Hydra from
> becoming overloaded with expensive queries from bots.

Very good point.

IME (which is a few years old at this point) at least the 
highlighted BingBot & SemrushThing always respected my robots.txt, 
but it's definitely a concern.  I'll leave this bug open to remind 
us of that in a few weeks or so…

If it does become a problem, we (I) might add some basic 
User-Agent sniffing to either slow down or outright block 
non-Guile downloaders.  Whitelisting any legitimate ones, of 
course.  I think that's less hassle than dealing with dynamic IP 
blocks whilst being equally effective here.

Thanks (again) for taking care of Hydra, Mark, and thank you Leo 
for keeping an eye on Cuirass :-)

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

      reply	other threads:[~2021-12-10 23:04 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-06 21:20 bug#52338: Crawler bots are downloading substitutes Leo Famulari
2021-12-06 22:18 ` bug#52338: [maintenance] hydra: berlin: Create robots.txt Leo Famulari
2021-12-09 13:27   ` bug#52338: Crawler bots are downloading substitutes Mathieu Othacehe
2021-12-09 15:42     ` Tobias Geerinckx-Rice via Bug reports for GNU Guix
2021-12-10 16:22       ` Leo Famulari
2021-12-10 16:47         ` Tobias Geerinckx-Rice via Bug reports for GNU Guix
2021-12-11  9:46           ` Mathieu Othacehe
2021-12-19 16:53             ` Mathieu Othacehe
2021-12-10 21:21 ` Mark H Weaver
2021-12-10 22:52   ` Tobias Geerinckx-Rice via Bug reports for GNU Guix [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=875yrw3vvk.fsf@nckx \
    --to=bug-guix@gnu.org \
    --cc=52338@debbugs.gnu.org \
    --cc=leo@famulari.name \
    --cc=me@tobias.gr \
    --cc=mhw@netris.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.