fedi/mastodon
1
0
Fork 1
mirror of https://github.com/mastodon/mastodon.git synced 2024-11-12 18:16:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update robots.txt to include extra bots and fix a UA typo

Browse source
This commit is contained in:
UmbreonShapeshifter 2024-08-17 10:14:28 +10:00 committed by GitHub
parent 1b97290155
commit efb70372c0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 7 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
public/robots.txt
View file

@ -4,6 +4,12 @@
User-agent: anthropic-ai
Disallow: /
User-agent: OmgiliBot
Disallow: /
User-agent: The Knowledge AI
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
@ -52,7 +58,7 @@ Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: 0AI-SearchBot
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot