robots.txt
Jonathan Davies
2005
0
This is the standard configuration for robots.txt to allow crawler all subpages of the whole site:
User-agent: *
Disallow:
