Exclude folders from search engine

23.12.2018
Skip to main content
< Zurück
Du bist hier:
Drucken

Exclude folders from search engine

Exclude folder/file from search engine

Create the file robots.txt according to Wikipedia:

http://de.wikipedia.org/wiki/Robots_Exclusion_Standard

# robots.txt for example.com 
# I'm ruling out those webcrawlers 
User agent: Sidewinder 
Disallow: /
 
User agent: Microsoft.URL.Control
Disallow: /
 
# These directories/files should not be searched
User agent: *
Disallow: /default.html
Disallow: /Temp/ # these contents will disappear soon
Disallow: /Private/Family/Birthdays.html

With the following commands the indexing of the complete web presence is forbidden to all search engines.

User agent: *
Disallow: //

Another example: robots.txt from Wikipedia

Tags:
FireStorm
Related Post