Verity Spider indexing navigation

[Microlight]

Hi there,

We are using Verity Spider on one of our client's site, and then use CFSEARCH
to return any results matching the user's search keyword.

One thing that our client has noticed is that, if the user searches for a word
that is used in the main navigation of the site, then some of the pages
returned are only there because that navigation link is there...If you do a
search on 'news' and a 'news' link appears on all pages, then all pages are
returned, dispite the fact that the word 'news' is only appearing in the
navigation!

Is there a known way of stopping the navigation of the site from being
indexed, but still followed?

Most of the navigation on that site is coming from <CFINCLUDE>s.

Many thanks,

Cheers

[Tom Jordahl]

If you are using vspider, it will index any text that is on the HTML page.

No way around it.

--
Tom Jordahl
Macromedia Server Development

[AndyMindcore]

I think I have a solution to your problem. You might need to tailor a few
things, but it should work. This is a general explanation as I was searching
for an answer to this very dilemma when I stumbled upon your post, and came up
with the below idea.

Specify a custom user agent name in your vspider command file or on the
command line:

-agentname "vspider indexing process"

In your site framework, make your navigation conditional on the user agent.
This way, you can strip it out if the user agent matches the one you set for
vspider. The pages might look like crap to vspider, but all you care about is
the content! This way, any "generic" sections of your pages are automatically
removed when your custom vspider agent crawls about at ungodly hours.

"But how do I get it to crawl the site if there's no navigation?" you might be
incredulously asking.

Create a "dummy" page for indexing that has your navigation on it, and specify
that links should be followed, but the page itself should not be indexed:

-indexclude "*/mydummylinkpage.cfm"

Of course, there are a few different ways to go about what I've outlined, but
I hope the above helps you out a bit.

Andy