Exclude folders from search engine

Support Wissensdatenbank

Du bist hier:

Exclude folder/file from search engine

Create the file robots.txt according to Wikipedia:

http://de.wikipedia.org/wiki/Robots_Exclusion_Standard

# robots.txt for example.com 
# I'm ruling out those webcrawlers 
User agent: Sidewinder 
Disallow: /
 
User agent: Microsoft.URL.Control
Disallow: /
 
# These directories/files should not be searched
User agent: *
Disallow: /default.html
Disallow: /Temp/ # these contents will disappear soon
Disallow: /Private/Family/Birthdays.html

With the following commands the indexing of the complete web presence is forbidden to all search engines.

User agent: *
Disallow: //

Another example: robots.txt from Wikipedia

War dieser Artikel hilfreich?
0 out of 5 stars
5 Stars 0%
4 Stars 0%
3 Stars 0%
2 Stars 0%
1 Stars 0%
Wie können wir diesen Artikel verbessern?
Brauchst du Hilfe?
Related Post