You do not have to be concerned that the result may throw up this "sitemap" page and would disappoint the visitor
. This will not happen, as the "site-map" has no searchable content and will not get included in the results, rather all other pages would. We found the site wired.com had published hierarchical sets of crawler pages. The first crawler page lists all the category headlines, these links lead to a set of links with all story headlines, which in turn lead to the news stories.