You probably know that Google finds its information by using "spiders" or "robots". These are pieces of software that continually tour the web looking for new, amended and deleted pages. However, there are some things that they look for to help them. One of these is the robots.txt file.
Since Google looks for this file, it makes sense to have one on your website, even though it is not mandatory. Spend 1 minute going through the following check to see whether you have one in place.
Go into your browser, and type your website address, followed by "/robots.txt". Note that robots.txt must be in lowercase. For example, http://www.google.com/robots.txt.
What happens? If you get an error page, then something is amiss. Hopefully you will get something displayed. It might look like gobbledygook, but it probably means something to the search engines.
This file is intended to tell search engines which files and folders you want to have included in or excluded from their search results. It may contain other guidance as well. However, it is not mandatory that these instructions are followed, so this is a poor way to apply "security".
The reason why we always look for a robots.txt file is that, if it is missing, then it gives an indication of the quality of search engine marketing that has been undertaken on a site. While a missing robots.txt it is not important in itself, SEO is all about attention to detail. So any such obviously missing detail doesn't bode well for the care taken with your online marketing.
If everything appears to be OK, then congratulations. Although this is how it ought to be, many websites fail this basic test.
But what if you have a problem? If you don't care about your website marketing, then it's not really an issue. However, you wouldn't have read this far, would you?
Usually I would suggest that you first contact whomever looks after your website. However, you may now realise that website design and website marketing are two different skill sets. That is why web designers pass clients to us. Get in touch and I can tell you how we could help you.
This is one of a series of articles that deal with common issues when promoting your business online. We hope that you found it useful. Go to our online marketing tips to see previous pieces of advice. If there are any topics that you would like covered, please let us know.
If you need to promote your business online, then contact us.