What Do Spiders,Robots Like and Dislike?

Certain parameters within our website are pleasing to spiders, robots and crawlers, by the way these terminologies are referring to search engines like Google, Yahoo etc. There are no physical spiders as such “crawling” around the net.

• One of the first files a spider visiting your website looks for is robot.txt (I will cover how to create and control this file at a future date). In it’s most basic form this is a file that has instructions on how a spider or crawler is to behave with your website,  in essence it has parameters that indicates to the search engine some of the following; When viewing the site is the bot allowed to do so in it’s entirety (all files) or partially or not at all, how frequently is the spider allowed to visit, the interval between visits etc. Many users are not aware that it is we who control when, where and how often our site can be visited by the search engine robots (except for the first time), not the other way around. If this file is missing on our site, then the robots visit and access our webpages and folders at their discretion. With this file we can place parameters within it to control the bot and how it reads and scans our website files. Not having this file has a neutral effect on your website.

• A site map is another of those files that the spider looks for. It uses this file to crawl and index other webpages within your website. A site map is basically a list of all your webpages. There are two forms of site maps; XML and HTML. The XML version is the one that search engines read and decipher, the HTML format is for your site visitors and users. This file is extremely important and needs to be constantly updated to the most current of your website. This is very similar to the index page you find at the back of a book. Some even refer to a site map as site index. I highly recommend you generate a site map for each of your website both the XML version and an HTML one. The XML is for the search engine bots and HTML for your site users. This is also a great SEO tool for your site. Also make sure that every time you make changes to your site (ie add, delete webpages) you generate a new site map which overwrites and replaces the old one.

• During the crawl of your website by a Bot, what impresses the search engines is dynamic fresh new original content. A site that has new content added on a regular basis is visited more often (unless of course you control it via the robot.txt file). A site with stagnant material does not get the same attention. A site with old, copied, duplicated material does not get a second glance, as a matter of fact in some cases repeated (copied) content or mirror websites (sites that have identical look and framework) are frowned upon by Google and can cost you demerit points. Most of these mirror sites are generated by affiliate associated websites. So if you wish a healthy ranking then continue to have original, fresh and frequently updated content. I am well aware of a situation where one website owners’ AdWords campaign got stopped by Google because he had an affiliate site that was a mirror site. There is nothing wrong or illegal with having mirror sites, but they are extremely hard to show up high on a SERP.

• Webpages that has content which matches up with their keywords, meta tags, title and description are received well by search crawlers. Rich content with good SEO techniques will get you good ranking. It takes time and some effort, so be patient and keep working at it.

• A site that pretends to show differently to Spiders and it’s visitors can get you banned. Many a user tries to use black hat SEO techniques to fool the Spiders and get ranked higher by pretending to be one kind of website, but are something entirely different. Don’t do it, do not try to fool the Search Engines it’s not worth it. The Algorithms are constantly tweaked and fine tuned to pick up on impostors and fraudulent websites. Be Warned. It could get your site into a heap of trouble.