How To Make Sure That Your Websites Are Fully Crawlable By The Search Engines

[Mitchell]

Making sure that your websites are fully crawlable is a key component for any search strategy, especially if you want your sites to rank well for multiple related keyword phrases or different products.

One of the ways that you can insure that a search engine spider crawls your site deeply is to provide it a road map of your site. This road map, called a "site map" is a very simple page (from a design standpoint) that only serves one purpose from a search engine optimizers point of view, to get those interior pages into a search engine's index.

A brief history of site maps . . .

Before site maps were recognized by the search engine optimization community as a tool for good search results, it was used by fortune 1000 companies to help visitors to their site find out where a particular service or product was located.

For example: If you want to do some sort of business with your bank online, you might find that their home page doesn't quite get you to the exact location on their site that you want.

After all, a typical bank might offer car loans, student loans, home loans, credit cards, investment accounts, mutual funds, etc.

If per chance you arrived at this banking site on a page other than the home page, you might get a little turned around from a site navigation standpoint.

So, site maps were created.

What is a site map?

A site map is a page that contains a basic HTML link to every page on that web site. Every page, not just your main topics but every page.

Site maps do not need to be fancy (in fact, it's better if they are not), they just need to contain a logical order and links to all of your pages.

How does this benefit us? Three ways . . .

First, a site map gives your customers an easy navigation system to every page in your web site. Now, don't confuse the use of a site map as a replacement for logical navigation on your regular pages. You want to make sure that your site can be navigated simply and easily from any page on your site.

However, some folks (a very small percentage) prefer to see the entire site's structure on one page and choose their destination from it.

Second, a site map is a fantastic way to get a search engine spider to see and crawl every single page in your site. When optimizing different pages in your web site for different keywords, a Site map is a perfect solution for ensuring that a spider can get to every one of those optimized pages.

Third, and this is big: A Site Map provides an opportunity to send link reputation to a particular page. Now, link reputation is a discussion that's beyond the scope of this aricle, but it is perhaps one of the most important factors in off Page search engine optimization.

You want the links on your Site Map Page to Say the right thing about the pages that they are linking to. It's like a vote. So, make sure that your site map is voting correctly for all of your interior pages.

Fourth, site maps are also an additional way to distribute Google page rank. If your site map is just one link off of the first page, it can pass a significant amount of Google page rank deep into your web site. This helps create a site structure where just about every internal page has the same chance of ranking as well as your home page.

Finally, site maps allow you to use dynamic linking strategies to control where page rank does and does not go throughout your site.

For example, a common mistake is to have a normal link to your privacy policy on every page of your site. In actuality, giving your privacy policy page the same page rank as every other page of the site is a big waste of whatever page rank you have available.

So use a good site map and you will reap the rewards.