INLS 490-154: Information Retrieval Systems Design and Implementation

Spring 2009. Thursdays 5:30-8:00pm, Greenlaw Hall 104

[ Home ] [ Syllabus ] [ Assignments ] [ Project ] [ Resources ]

Assignment-8: Evaluation
Assigned on: 03/05/2009, Due on: 03/17/2009
1. Build an index using Krovetz stemmer and stop words removal, taking the TREC file given from the LA Times (with over 130,000 documents). Run topic 402 with its title, description, and narrative parts as queries retrieving up to 100 documents using TFIDF retrieval model. Calculate (1) recall and precision at different recall points (at ranks 5, 10, 15, 20, 30, 100, 200, 500, and 1000) for each query, (2) MAP, (3) GMAP, (4) R-precision for each query, (5) bpref for each query, (6) reciprocal rank for each query, and (7) MRR using 'trec_eval'. [7 points]
2. Build an index using Porter stemmer and stop words removal, taking the first 10 countries' descriptions from the CIA World Factbook as documents. Use independence as a query and rank all the documents (or as many as you can) using TFIDF as well as Okapi retrieval models. Compare these rank-lists using (1) Pearson's covariance, (2) Spearman's Rho test, and (3) Kendall's Tau test. [3 points]
What can you say based on these values? Give your comments in 2-4 sentences. [2 points]

| Chirag Shah | Last update: May 3, 2009 |