Professional Web Applications Themes

Verity Spider indexing navigation - Coldfusion - Advanced Techniques

Hi there, We are using Verity Spider on one of our client's site, and then use CFSEARCH to return any results matching the user's search keyword. One thing that our client has noticed is that, if the user searches for a word that is used in the main navigation of the site, then some of the pages returned are only there because that navigation link is there...If you do a search on 'news' and a 'news' link appears on all pages, then all pages are returned, dispite the fact that the word 'news' is only appearing in the navigation! Is ...

  1. #1

    Default Verity Spider indexing navigation

    Hi there,

    We are using Verity Spider on one of our client's site, and then use CFSEARCH
    to return any results matching the user's search keyword.

    One thing that our client has noticed is that, if the user searches for a word
    that is used in the main navigation of the site, then some of the pages
    returned are only there because that navigation link is there...If you do a
    search on 'news' and a 'news' link appears on all pages, then all pages are
    returned, dispite the fact that the word 'news' is only appearing in the
    navigation!

    Is there a known way of stopping the navigation of the site from being
    indexed, but still followed?

    Most of the navigation on that site is coming from <CFINCLUDE>s.

    Many thanks,

    Cheers

    Microlight Guest

  2. #2

    Default Re: Verity Spider indexing navigation

    If you are using vspider, it will index any text that is on the HTML page.

    No way around it.

    --
    Tom Jordahl
    Macromedia Server Development


    Tom Jordahl Guest

  3. #3

    Default Re: Verity Spider indexing navigation

    I think I have a solution to your problem. You might need to tailor a few
    things, but it should work. This is a general explanation as I was searching
    for an answer to this very dilemma when I stumbled upon your post, and came up
    with the below idea.

    Specify a custom user agent name in your vspider command file or on the
    command line:

    -agentname "vspider indexing process"

    In your site framework, make your navigation conditional on the user agent.
    This way, you can strip it out if the user agent matches the one you set for
    vspider. The pages might look like crap to vspider, but all you care about is
    the content! This way, any "generic" sections of your pages are automatically
    removed when your custom vspider agent crawls about at ungodly hours.

    "But how do I get it to crawl the site if there's no navigation?" you might be
    incredulously asking.

    Create a "dummy" page for indexing that has your navigation on it, and specify
    that links should be followed, but the page itself should not be indexed:

    -indexclude "*/mydummylinkpage.cfm"

    Of course, there are a few different ways to go about what I've outlined, but
    I hope the above helps you out a bit.

    Andy

    AndyMindcore Guest

Similar Threads

  1. Verity Spider license
    By benwilcox in forum Coldfusion - Getting Started
    Replies: 0
    Last Post: September 28th, 06:24 PM
  2. Verity Spider on Apache Virtual Hosts
    By UniWebDude in forum Coldfusion Server Administration
    Replies: 0
    Last Post: May 6th, 10:34 AM
  3. ColdFusion MX 6.1 and Verity Spider
    By Belluz in forum Coldfusion - Advanced Techniques
    Replies: 5
    Last Post: April 11th, 02:28 PM
  4. Verity Spider Issues, CFMX7
    By YAK in forum Coldfusion - Advanced Techniques
    Replies: 4
    Last Post: March 24th, 11:00 PM
  5. Question regarding Verity Spider
    By Microlight in forum Macromedia ColdFusion
    Replies: 0
    Last Post: March 21st, 03:11 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139