Naked Science Forum

Non Life Sciences => Geek Speak => Topic started by: engrByDayPianstByNight on 30/07/2008 17:07:58

Title: How to develop a good methodology to test new search engine (e.g., cuil)?
Post by: engrByDayPianstByNight on 30/07/2008 17:07:58

When cuil (cuil.com) was released earlier this week, I tried it, and frankly, was not very impressed with its capabilities. It has made some mistakes when searching for my queries, and sometimes, it wouldn't return any hits at all, even though the same string of keywords returned lots of hits when using google. So, cuil still has a few kinks to be worked out.

Anyways, I'm curious if anybody (or better yet, someone who actually works at google) can share with us some idea on how to systematically test the underlying strengths and weaknesses of a new search engine? The way I did it with cuil,I  randomly chose some key words and compare the search results with google. Fortunately, the current state of cuil is bad enough for me to realize it's still got problems. But suppose if it were pretty good, how then should I be testing its performance?

Any ideas? Thanks.

Title: How to develop a good methodology to test new search engine (e.g., cuil)?
Post by: engrByDayPianstByNight on 02/08/2008 00:43:36

Well, maybe I'll rephrase my question. How would you guys go about evaluating a new search engine? For example, testing your favorite keyword string and counting the number of hits returned, whether the top hits match what you want (i.e., some intuitivity of the search engine in figuring out what you might be looking for), etc.?

Title: How to develop a good methodology to test new search engine (e.g., cuil)?
Post by: DoctorBeaver on 02/08/2008 07:50:58

I think all you can do is compare the hits with those you get from other search engines.

Title: How to develop a good methodology to test new search engine (e.g., cuil)?
Post by: engrByDayPianstByNight on 03/08/2008 20:44:56

Yeah, I guess that's probably what an average user would do. I'm just curious there has to be some kind of benchmark metrics we could use to evaluate the performance of a new search engine and claim whether it's superior to the existing ones. Oh well. Thanks, doc.