Rewrite robots.txt

Your CMS (read: TYPO3) can handle multiple subdomain in one installation. Problem: you can only put one robots.txt in the htdocs. The following rewrite rule will solve the problemby rewriting robots.txt to robots_(subdomain).txt

1
2
3
4
5
6
7
RewriteCond %{HTTP_HOST} ^([^.]+)\.domain\.tld [NC]
RewriteRule robots\.txt$ robots_%1.txt [L]
 
# default robots.txt
RewriteCond %{REQUEST_FILENAME} robots(_.*).txt$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule robots(_.*).txt$ robots_def.txt [L]

it will do following rewrite:

  • www.domain.tld/robots.txt will load robots_www.txt
  • sub1.domain.tld/robots.txt will load robots_sub1.txt, if it’s not exist, then robots_def.txt will be loaded