By: Lukene

Lukene — Mon, 01 Feb 2010 00:29:53 +0000

Robots.txt
Robots.txt file does not improve your search engine positioning.
It provides robots with information concerning which files you will not allow to be crawled and indexed in the search engines.
When the search engine robot crawls your site it looks for the robots.txt file.
If it doesn’t find one it assumes automatically that it may crawl and index the entire site.
This allows all robots to crawl all files.
User-agent: *
Disallow:
This Disallows all robots to crawl a folder called /cmsbuffet/ .
User-agent: *
Disallow: /cmsbuffet/
This web site validates your Robots.txthttp://tool.motoricerca.info/robots-chec…

By: SZ Quadri

SZ Quadri — Sun, 31 Jan 2010 18:49:16 +0000

robots.txt file is just a simple text (notepad kind) file that defines how the crawlers should crawl your web site. It is a kind of advisory that tells bots that a particular folder (or the complete web site) is to be crawled or not.
It appears that the default robots.txt file that you have on your web site is telling the bots that your web site not to be crawled.
Create a text file named robots.txt and paste the below given text into it then save it and upload to root of your web site. Your blogging software control pabel should have a provision to update or upload the robots.txt file.
User-agent: *
Disallow:
The above tells the bots that everything is to be crawled by every crawler/bot.
SZ Quadri

Comments on: Robot.txt File??

By: Lukene

By: SZ Quadri