In the view of Raul Valdes-Perez, CEO of search engine Vivisimo, most search engines aren’t really doing their job.
The “ranking engines”, as he calls them, fall down because they offer up undifferentiated page results ranked by popularity, freshness and links—criteria that don’t do enough to make search results useful to searchers. The product of these searches is long lists of URLs served up indiscriminately. “It’s like going into a bookstore and finding all the books lying on the floor in piles,” he says. “But you don’t see that in a bookstore: Things are categorized and organized on shelves. We want to transform the way people view information in the electronic world, moving away from the disorganized lists of the ranking engines toward something more meaningful.”
Valdes-Perez set that information transformation in motion in 1999 when, as a computer science professor at Carnegie Mellon University, he co-founded Vivisimo with some research colleagues and began looking for a better, more organized way to search.
After a few years of development and initial investments from both the National Science Foundation and the Commonwealth of Pennsylvania, the Pittsburgh-based company brought out a metasearch platform that queries different search engines and clusters results into dynamic folders. The idea is to give users a better sense of the scope of their query by showing the various categories into which results fall, some relevant but others not. For example, a search for “Nike” organizes the results into folders not only for the shoe and its manufacturer—which occupy the first two results pages of a Google search on the word-- but also for the Greek goddess and the ballistic missile.
The result is a broader, deeper view of the search world at a single glance. “When people search with ranking engines, they look at the first five results, and then they go away,” says Valdes-Perez. “So the decisions they make using those results are based on a very limited view of the world. Clustering those results allows you to make decisions based on one hundred times more information from the Web, and in the same amount of time. You can quickly find a result that would be ranked down at number 133 by the ranking engines, and it might be just the one you need. So clustering makes you smarter faster.”
That’s one reason Vivismo’s enterprise technology, called Velocity, has been adopted by government intelligence agencies such as the National Security Agency, the Defense Intelligence Agency, and a German intelligence body. The same goes for government Web sites such as NASA and corporate users such as Cisco Systems and Procter & Gamble, which have deployed the Velocity Suite behind their intranets. The Journal of the American Medical Association, the Institute of Physics, and the American Association for the Advancement of Science have all found use for Vivisimo’s way of gathering and sorting Web knowledge.
Finally, in September 2004, Vivisimo opted to give the non-spies and un-academics among us a shot at the same smarts: The company launched Clusty, a beta version of a Web-based search using the same clustering technology running under its Velocity product. The Clusty search is outwardly similar to the late, lamented Northern Light search engine, which discarded its free Web-based model in 2002 for a paid enterprise one. But the important difference is under the hood. Northern Light’s categories—its “taxonomy”—were pre-defined by humans. “They had librarians carving up the world of their own index and refining that carving over time,” says Valdes-Perez. “They weren’t doing metasearch, which involves constantly updating your taxonomy for new classes of content. For that reason, they couldn’t scale and maintain their business model.”
By comparison, Clusty searches the Web by going out to about a dozen different sources, some of them proprietary to Vivisimo but others general-purpose Web crawlers. Then the clustering technology is applied to generate spontaneous categories automatically, based on the specific results acquired.
Users can conduct searches from the Web site or download the Clusty toolbar, which offers additional functionality such as the ability to highlight a search term on a Web page, and “Query Catcher”, which fills in terms you’re searching for on Google, Yahoo or other engines. The idea is that if you don’t find what you’re looking for in the URL lists produced by those searches, you can see if Clusty does better with just one button click.
And Valdes-Perez is confident it will do better. “Just go to Clusty, type in a search term and see how much you can learn about that topic in one minute compared to Google or Yahoo,” he says. “If they try it, people get it. After this, using a ranking engine is like riding a bicycle without handlebars.”
More people are about to get it, thanks to a deal Vivisimo signed to provide its clustering technology to the revamped AOL Search. That’s a big, high-visibility win for a technology that has slowly been acquiring fans behind the scenes. “That’s our most significant deal so far, and it will really expand the reach of this emerging clustering technology,” Valdez-Perez says.
Whatever AOL decides to do to monetize its new search, Vivisimo has no plans to make money off Clusty. Right now, Clusty searches show paid-placement ads from Overture, but Valdes-Perez says the company won’t offer its own paid ad programs. Instead, Vivismo will continue to earn by licensing its technology to others. “We aim to be the ‘Intel Inside’ of Web search,” he says. “Clusty is our personal expression of what we think searching should look like on the Web and within the enterprise.”