New robots.txt to allow DDG
This commit is contained in:
parent
51e6576b8a
commit
2338fdcafa
|
@ -1,2 +1,11 @@
|
|||
User-agent: DuckDuckBot
|
||||
Disallow: /cursors/
|
||||
Disallow: /images/
|
||||
Disallow: /fonts/
|
||||
Disallow: /moderator/
|
||||
Disallow: /login
|
||||
Disallow: /logout
|
||||
Disallow: /human/
|
||||
|
||||
User-agent: *
|
||||
Disallow: /
|
||||
|
|
Loading…
Reference in a new issue