robots.txt

Jonathan Davies 1406 0

robots.txt

This is the standard configuration for robots.txt to allow crawler all subpages of the whole site:

User-agent: *
Disallow:

Tags:

Previous Article

Reasonable User Configuration in .bashrc

Next Article

robots-make-pages

What's Your Reaction?

0

0

0

0

0

0

0

Jonathan Davies

After having accumulated thousands of valuable notes in private over the years, I decided to share them on this webpage and make them public for the benefit and improved work efficiency of others. Good luck.

Related Posts

Overview and comparison over the different Javascript libraries for 2021

Overview and comparison over the different Javascript libraries...

Jonathan Davies 0 2209

Example pages made with Boostrap

Example pages made with Boostrap

Jonathan Davies 0 1543

jQuery - Introduction & possibilities

jQuery - Introduction & possibilities

Jonathan Davies 0 1666

Fontawesome

Jonathan Davies 0 1553

HTML 5 - Overview tags

HTML 5 - Overview tags

Jonathan Davies 0 1531

Overview over the most efficient Bootstrap 4 Classes

Overview over the most efficient Bootstrap 4 Classes

Jonathan Davies 0 1553

Comments