Mirror
/
tildes
mirror of https://gitlab.com/tildes/tildes.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Browse Source

robots.txt: Block OpenAI GPTBot

master
Deimos 9 months ago
parent
d0d6b6d3dc
commit
f355bc6a80
1 changed files with 4 additions and 0 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 4
      tildes/static/robots.txt

4
tildes/static/robots.txt
View File

@ -30,6 +30,10 @@ Disallow: /
User-agent: dotbot User-agent: dotbot
Disallow: / Disallow: /
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /
# Unknown/suspicious scraper - UA only contains a gmail address # Unknown/suspicious scraper - UA only contains a gmail address
User-agent: MauiBot User-agent: MauiBot
Disallow: / Disallow: /

Write Preview
Loading…
Cancel
Save