It seems like every week I get an email asking about what a robots.txt file is and how to set one up for Magento 1.x so I put together this little guide to get you started.
Firstly, it is important to remember that a robots.txt is only obeyed by good robots/crawlers and not malicious ones. So don’t look at a robots.txt file as a security measure, instead look at it like adding signs to doors inside building saying “Employees Only” or “Keep Out”. You still need to remember to actually lock the doors to keep the dishonest people out. In short, a robots.txt tells search engines where they are allowed to look and where they should ignore for search engine optimization purposes only!
The robots.txt file is located in the root of your Magento 1.x store, if it isn’t there you can create the file yourself.
# SEF and Customer URLS
# Miscellaneous files.
# Remove Filtered URLs
## Disallow session IDs
Once you have created/updated your robots.txt you will want to include any frontend Ajax controllers from third party extensions.
You should also never in anyway include the URL for your store’s administration panel in your robots.txt for security reasons.