robots.txt

Jonathan Davies 1961 0

robots.txt

This is the standard configuration for robots.txt to allow crawler all subpages of the whole site:

User-agent: *
Disallow:

Tags:

Previous Article

Reasonable User Configuration in .bashrc

Next Article

robots-make-pages

What's Your Reaction?

0

0

0

0

0

0

0

Jonathan Davies

After having accumulated thousands of valuable notes in private over the years, I decided to share them on this webpage and make them public for the benefit and improved work efficiency of others. Good luck.

Related Posts

Example pages made with Boostrap

Example pages made with Boostrap

Jonathan Davies 0 2100

Overview and comparison over the different Javascript libraries for 2021

Overview and comparison over the different Javascript libraries...

Jonathan Davies 0 3065

Varient CMS

Jonathan Davies 0 2313

Fontawesome

Jonathan Davies 0 2124

Bootstrap

Jonathan Davies 0 2336

Schrift & Schriften für ein Webprojekt

Schrift & Schriften für ein Webprojekt

Jonathan Davies 0 2426

Comments