If you're into search marketing you'll have heard about the search engine robots, also known as crawlers or spiders. These automated software bots are the primary way that search engines read the content of your site and populate their index with results that map against specific keywords.

Robot activity can get pretty arcane but if you're a marketer, there's a simple rule to observe: If a robot doesn't see the pages on your site, you'll never be listed in the search results. Even if your efforts are focused on PPC search marketing, you should try and get listed for a subset of your best search terms. Getting listed in the engines means they have to crawl your site, and that means you have got to know the robots and their behaviour.

What You Need To Know

The fact that a robot has ‘visited' your site may be interesting, but it isn't meaningful by itself. Why? Because knowing that a robot has visited isn't enough to be able to predict whether or not you'll actually get listed. What does matter is:

• Frequency: How often the robot visits
• Recency: The time/date of the robot's last visit
• Pages seen: Which specific pages were visited by the robot

The frequency and recency of information are important because they give you an idea of when your site was last crawled, and how much time 'typically' passes between crawls. As a benchmark, it's helpful to know that most search engines visit a site monthly. Very popular sites (ClickZ.com, for example) get crawled more often, sometimes many times a day. Knowing frequency and recency can help you predict when the robot is likely to revisit, helping you time your improvements accordingly.

Which Pages Does The Robot See?

A robot crawls a web site one page at a time, in no particular order. But a robot isn't like a nosy party guest, poking around all of your site's nooks and crannies. It's much more demure—it will ONLY visit pages it can get to via a simple HREF link. That's why it's so important to notice sooner, rather than later, pages that aren't being crawled. Adding a simple HREF link to the important pages in your site can save you hours of time and loads of lost traffic down the road.

All Robots Are Not Alike…Except When They're Exactly The Same

All robots are not alike. Robots for different search engines behave in very different ways—some visit much more frequently than others, and some don't visit at all.

You'll also notice in some cases, that robots aren't just 'acting' alike—they're the exact same! If you monitor robot activity, you will notice that Ask Jeeves and Teoma share the same robot.