Robot.txt File??
Posted by Adsense Enquirer on January 31st, 2010 at 02:13pm
I signed up for Google Adsense on my blog, and it said the crawlers blocked my site because I need a robot.txt file to show relevant ads on my profile. What is this robot.txt file? I have never heard of it. Does this mean I will not make money from while the crawlers are blocked? I am new to this stuff can someone help?
Under adsense money
2 Comments for Robot.txt File??
1. SZ Quadri | January 31st, 2010 at 6:49 pm
robots.txt file is just a simple text (notepad kind) file that defines how the crawlers should crawl your web site. It is a kind of advisory that tells bots that a particular folder (or the complete web site) is to be crawled or not.
It appears that the default robots.txt file that you have on your web site is telling the bots that your web site not to be crawled.
Create a text file named robots.txt and paste the below given text into it then save it and upload to root of your web site. Your blogging software control pabel should have a provision to update or upload the robots.txt file.
User-agent: *
Disallow:
The above tells the bots that everything is to be crawled by every crawler/bot.
SZ Quadri
2. Lukene | February 1st, 2010 at 12:29 am
Robots.txt
Robots.txt file does not improve your search engine positioning.
It provides robots with information concerning which files you will not allow to be crawled and indexed in the search engines.
When the search engine robot crawls your site it looks for the robots.txt file.
If it doesn’t find one it assumes automatically that it may crawl and index the entire site.
This allows all robots to crawl all files.
User-agent: *
Disallow:
This Disallows all robots to crawl a folder called /cmsbuffet/ .
User-agent: *
Disallow: /cmsbuffet/
This web site validates your Robots.txthttp://tool.motoricerca.info/robots-chec…
Leave a Comment for Robot.txt File??
Trackback this post | Subscribe to the comments via RSS Feed