The High Bar For Relevancy?
A big chunk of the billions that go to search-engine marketing and search engine optimization
, SEM and SEO, (mostly to you-know-who) are spent on getting to Page 1 of the results.
I wont be the first to point out that relevance for in-house search i.e., without using Pagerank is a harder nut to crack. How much harder? A recent study from Aberdeen Group, publicized this week in Information Week, provides the following stat:
At top performing companies [defined as "Best in Class", the top 20% of those surveyed], 67% of searches returned the most relevant results on the first search results page, while lower rated companies saw relevant results on the first page for only 42% of searches.
1 out of 3 searches at best dont deliver the right search on the first results page. In other words, the best case for search=find is 67%.
Relevancy is as much art as science; the best solutions for the problem are the ones that provide a way to match the art to the science. If you need some background on relevancy, read the seminal article on relevancy and findability by Grant Ingersoll, and check out the fine presentation on the subject by Mark Bennett of New Idea Engineering delivered at the most recent SFBay Lucene/Solr Meetup we co-sponsored with the Computer History Museum in early September.
One of the best implementations of findability in Lucene and Solr Ive come across is at Netflix. Theres a really nice discussion captured in some slides by Walter Underwood, who helped built the Solr search infrastructure at Netflix (a milestone in a very distinguished career in search). He gave a terrific presentation at that same most recent Meetup.
A key metric Walter used at Netflix to gauge finding (search relevancy effectiveness is such a mouthful) is called Mean Reciprocal Rank, or MRR. Simply put, it gives one point for a click through to the first-ranked item, 1/2 a point to the second ranked item, 1/3 of a point to the 3rd ranked, etc. While it may not help find relevancy bugs, it provides a very nice aggregate picture of users experience finding what they look for. A good benchmark, or stretch goal, according to Walter: 0.5 MRR, with 85% of clicks on #1.
Let me be quick to say that there is much that is unique about the Netflix search use case (and much that is really, really fun). But the contrast between 85% of results selected at #1 in the results ranking, vs. 2/3 of results on the first page at best in class enterprise search implementations, leads me to wonder: what are others doing to measure relevancy and programmatically build feedback loops, automatic or otherwise? Lucene and Solr provide transparent, rich interfaces for doing this; and according to the Aberdeen study, theres plenty of opportunity to do so.
One Response to The high bar for relevancy?
1.Netflix may be an extreme case of known-item search, where people are always looking for the title (or a person) instead of a class of things. In addition, the studios spend millions of dollars teaching those titles (the search terms) to people. That doesnt mean that people can spell Ratatouille or Coraline, but it helps.
At a site selling cordless drills or fleece jackets, I seriously doubt you would get 85% of clicks on the first result. That would be dominated by informational searches, not known-item, known-title searches.
If your site deals with mass-market artistic works (books, film, music), the searches might look more like Netflixs.
Thanks for the writeup, it was a fun meetup with lots of good presentations. The Babbage engine is amazing, too.
by: Lucid Imagination
Choosing Seniorfacility.com For Their Important Old Time Is The Right Decision For Senior Personalit Find Out The Truth That Lies Deep Down on Six Feet Under Episodes What Are The Odds That You Will Win? How To Become A Busy Entrepreneurial Mom Steve Carell Leaving the Office - What Will Happen? Opt for Buses from Goa to Mumbai for Hassle free Journey Bichon Frise - Facts You Must Know Before Adopting Bichon Frise Does Cobroxin Have Any Side Effects? Utah Ready for the Big Time Things To Do In Malaysia Times Of Captain America Do What You Do Best MDX debuggers - How they help simplify your query process