all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#52338: Crawler bots are downloading substitutes
@ 2021-12-06 21:20 Leo Famulari
  2021-12-06 22:18 ` bug#52338: [maintenance] hydra: berlin: Create robots.txt Leo Famulari
  2021-12-10 21:21 ` Mark H Weaver
  0 siblings, 2 replies; 10+ messages in thread
From: Leo Famulari @ 2021-12-06 21:20 UTC (permalink / raw)
  To: 52338

I noticed that some bots are downloading substitutes from
ci.guix.gnu.org.

We should add a robots.txt file to reduce this waste.

Specifically, I see bots from Bing and Semrush:

https://www.bing.com/bingbot.htm
https://www.semrush.com/bot.html




^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-12-19 16:55 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-12-06 21:20 bug#52338: Crawler bots are downloading substitutes Leo Famulari
2021-12-06 22:18 ` bug#52338: [maintenance] hydra: berlin: Create robots.txt Leo Famulari
2021-12-09 13:27   ` bug#52338: Crawler bots are downloading substitutes Mathieu Othacehe
2021-12-09 15:42     ` Tobias Geerinckx-Rice via Bug reports for GNU Guix
2021-12-10 16:22       ` Leo Famulari
2021-12-10 16:47         ` Tobias Geerinckx-Rice via Bug reports for GNU Guix
2021-12-11  9:46           ` Mathieu Othacehe
2021-12-19 16:53             ` Mathieu Othacehe
2021-12-10 21:21 ` Mark H Weaver
2021-12-10 22:52   ` Tobias Geerinckx-Rice via Bug reports for GNU Guix

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.