Navigation and the Search Engines

Just being listed in the search engines isn't good enough - you want to try to rank as high as possible in searches. This means optimizing the appropriate keywords, and your navigational scheme can be a part of this.

Some search engines will assign more weight to a keyword that appears as a heading than one that is plain body text. So you may want to make sure that your section headings contain plenty of keywords. This may have a chilling effect on creativity - no more cute chapter titles - but everybody's doing it. You can't prevent the English language from going to Hell (it's already crossed over the Styx), so you may as well profit from its demise.

We are also told that some search engines assign more weight to keywords that appear as hyperlinks, or even as file names. So make sure that there are plenty of links to the sections that you want people to find, and that they are clearly labeled. Of course, well-thought-out navigational schemes will do this anyway. Can it be that the file name really matters? To some search engines it does, so make your file names descriptive of the page content (no, don't name all your files "sex_sex_sex.html").

This brings up a point that a lot of people seem to miss - the point is not simply to bring as many people as possible to your site any way you can. The point is to bring the right kind of people to the pages that you really want to be seen. If you're selling widgets, then you want widget collectors, not sex-starved adolescents (research shows that adolescents buy few widgets). So bulking up on lascivious keywords will bring you quantity, but not quality. Sales and return visits are what you're after, not just raw traffic. Choose keywords that describe whatever it is you're selling, as descriptively and specifically as possible.

For the latest news on how the various search engines rank sites, see
Search Engine Watch.

Also worth thinking about is the question of which pages tend to come up in searches. Generally speaking, you want first-time visitors to your site to enter through the home page or one of the main section hubs, not through minor content pages. Let's say someone searches for widgets. If your home page comes up at the top of the search, that's good, because there they can see the whole range of widget-related content that you have, and are likely to explore your site. If on the other hand, the page that comes up is a minor page three levels down, which explains how to install some esoteric attachment to one of your many models of widget, they are likely to get confused and split. If your site has a lot of pages, then pages of minor importance can easily crowd out your home page altogether in search results.

This problem exists because automated spiders blindly count keywords. They have no way to know which pages are really the most relevant, although techniques such as assigning extra weight to links and/or headings, as described above, are an attempt to make them smarter in this regard. Therefore, you should make sure that every single page on your site includes a navbar with links to your main sections and your homepage. No matter what you do, some people will enter by the bathroom window, so make sure they can easily find their way to the front door.

Another way to address this problem is to use a robots.txt file to try to exclude certain pages from search engines altogether. In a recent article in The Web Developer's Journal ( 10 Ways Not to Promote Your Web Site), I explained how to set up a robots.txt file, which tells visiting spiders not to index certain pages. Any pages which are not meant to be seen by the public (templates, "test" pages, etc.) should definitely be included in this file. If you use frames, then all pages which represent the content of frames (in other words, all pages that are not framesets) should be here as well. You may also want spiders to exclude pages of marginal significance, such as disclaimers, copyright info, etc.

In extreme cases, some sites want to make sure that people enter only through the home page. They use the robots.txt file to exclude every page other than the home page. Of course, sites with lots of pages of loosely-related content, or sites that sell a lot of different products, should take the opposite approach. The faster someone can find exactly what they want, the better, so every content page can and should be thought of as an entry page in its own right. Analyze your log files to find out which pages are your main entry pages (see There's gold in them there log files! ) and adjust things accordingly.

Sensible Navigation
Build Traffic Through Good Site Design
Maximize Traffic with Internal Links