robots.txt
Jonathan Davies
1664
0
This is the standard configuration for robots.txt to allow crawler all subpages of the whole site:
User-agent: *
Disallow:
