Evaluation Fundamentals and Interleaving Techniques in Information Retrieval

Slide Note
Embed
Share

Explore the essentials of evaluation in Information Retrieval, covering topics like test collections, ranking, interleaving, and user studies. Understand batch and interleaved evaluation methods, user interaction feedback, and the motivation behind practical online retrieval evaluation. Dive into balanced interleaving with a focus on comparing and selecting ranked lists, illustrated with a case study on Napa Valley rankings.

sha_wh
sha_wh Follow

Uploaded on Sep 26, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

  1. Evaluation INST 734 Module 5 Doug Oard

  2. Agenda Evaluation fundamentals Test collections: evaluating sets Test collections: evaluating rankings Interleaving User studies

  3. Batch Evaluation Query Search Ranked List

  4. Interleaved Evaluation Query Search Ranked List Selection Document

  5. User Studies Query Formulation Query Search Ranked List Query Reformulation and Relevance Feedback Selection Document Examination Document

  6. Interleaving Combine two result sets Balanced or Team-Draft Avoid near-duplicates Assign credit as people click Averaging by session or by query Prefer the system that generates more clicks

  7. Motivation: A Taste Test Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013

  8. Balanced Interleaving Ranking A Ranking B 1. 2. 3. 4. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 5. Napa Valley Wineries and Wine www.napavintners.com 6. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley Napa Valley The authority for lodging... www.napavalley.com Napa Valley Wineries - Plan your wine... www.napavalley.com/wineries Napa Valley College www.napavalley.edu/homex.asp 1. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 2. Napa Valley The authority for lodging... www.napavalley.com 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 5. NapaValley.org www.napavalley.org 6. The Napa Valley Marathon www.napavalleymarathon.org Presented Ranking Napa Valley The authority for lodging... www.napavalley.com 2. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 3. Napa Valley Wineries Plan your wine... www.napavalley.com/wineries 4. Napa Valley College www.napavalley.edu/homex.asp 5. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 6. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 7. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 1. Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013 [Joachims 02]

  9. Balanced Interleaving Winner! Ranking A Ranking B 1. 2. 3. 4. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 5. Napa Valley Wineries and Wine www.napavintners.com 6. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley Napa Valley The authority for lodging... www.napavalley.com Napa Valley Wineries - Plan your wine... www.napavalley.com/wineries Napa Valley College www.napavalley.edu/homex.asp 1. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 2. Napa Valley The authority for lodging... www.napavalley.com 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 5. NapaValley.org www.napavalley.org 6. The Napa Valley Marathon www.napavalleymarathon.org Presented Ranking Napa Valley The authority for lodging... www.napavalley.com 2. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 3. Napa Valley Wineries Plan your wine... www.napavalley.com/wineries 4. Napa Valley College www.napavalley.edu/homex.asp 5. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 6. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 7. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 1. [ Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013 [Joachims 02]

  10. Team Draft Interleaving Ranking A Ranking B 1. 2. 3. 4. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 5. Napa Valley Wineries and Wine www.napavintners.com 6. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley Napa Valley The authority for lodging... www.napavalley.com Napa Valley Wineries - Plan your wine... www.napavalley.com/wineries Napa Valley College www.napavalley.edu/homex.asp 1. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 2. Napa Valley The authority for lodging... www.napavalley.com 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 5. NapaValley.org www.napavalley.org 6. The Napa Valley Marathon www.napavalleymarathon.org Presented Ranking Napa Valley The authority for lodging... www.napavalley.com 2. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Wineries Plan your wine... www.napavalley.com/wineries 5. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 6. Napa Valley College www.napavalley.edu/homex.asp 7 NapaValley.org www.napavalley.org 1. A B Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013 [Radlinski et al. 2008]

  11. Team Draft Interleaving Ranking A Ranking B 1. 2. 3. 4. Been There | Tips | Napa Valley www.ivebeenthere.co.uk/tips/16681 5. Napa Valley Wineries and Wine www.napavintners.com 6. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley Napa Valley The authority for lodging... www.napavalley.com Napa Valley Wineries - Plan your wine... www.napavalley.com/wineries Napa Valley College www.napavalley.edu/homex.asp 1. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 2. Napa Valley The authority for lodging... www.napavalley.com 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 5. NapaValley.org www.napavalley.org 6. The Napa Valley Marathon www.napavalleymarathon.org Presented Ranking Napa Valley The authority for lodging... www.napavalley.com 2. Napa Country, California Wikipedia en.wikipedia.org/wiki/Napa_Valley 3. Napa: The Story of an American Eden... books.google.co.uk/books?isbn=... 4. Napa Valley Wineries Plan your wine... www.napavalley.com/wineries 5. Napa Valley Hotels Bed and Breakfast... www.napalinks.com 6. Napa Balley College www.napavalley.edu/homex.asp 7 NapaValley.org www.napavalley.org 1. Tie! Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013 [Radlinski et al. 2008]

  12. Comparison with Expert Judgments [Radlinski & Craswell 2010; Chapelle et al. 2012] Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013

  13. Getting Clicks Build your own search engine for some special collection Direct access to index, generate any rankings Build on top of a public search API Fetch results from a search API, build your own results page Use a Web proxy or browser toolbar Intercept and modify the page the browser gets Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013

  14. Practicalities Easy to get users Easy on/off? Easy to observe Runs on Amount of work Changes are easy Method Robust All web traffic Proxy server Toolbar Everything client Queries & clicks Search API server Search engine Queries & clicks server Credit: Filip Radlinsky and Katja Hofmann, Practical Online Retrieval Evaluation, 2013

  15. Agenda Evaluation fundamentals Test collections: evaluating sets Test collections: evaluating rankings Interleaving User studies

Related


Intelligent Information Retrieval: Models, Ranking, and Algorithms

Intelligent Information Retrieval: Models, Ranking, and Algorithms

layl_2
Enhancing Information Retrieval with Augmented Generation Models

Enhancing Information Retrieval with Augmented Generation Models

pranav
Ranked Retrieval Models in Information Retrieval

Ranked Retrieval Models in Information Retrieval

zadki
Understanding Retrieval Models in Information Retrieval

Understanding Retrieval Models in Information Retrieval

calv_92
Strategies for Improving Memory and Revision Techniques for GCSE Success

Strategies for Improving Memory and Revision Techniques for GCSE Success

bronson
Introduction to Information Retrieval and Web Search

Introduction to Information Retrieval and Web Search

miley
Introduction to Information Retrieval and Web Search

Introduction to Information Retrieval and Web Search

wonderly_k
Challenges and Innovations in CXL 3.0 Dynamic Capacity Devices

Challenges and Innovations in CXL 3.0 Dynamic Capacity Devices

jaquely
Understanding Probabilistic Information Retrieval: Okapi BM25 Model

Understanding Probabilistic Information Retrieval: Okapi BM25 Model

sabo_hid
Enhancing Geography Learning Through Metacognitive Retrieval Practice

Enhancing Geography Learning Through Metacognitive Retrieval Practice

monica
Bayesian Methodology for Soil Parameters Retrieval from SAR Images

Bayesian Methodology for Soil Parameters Retrieval from SAR Images

jaiy_607
Understanding Core Concepts in Information Retrieval: Lexical and Semantic Gaps, Retrieval Models, and Algorithms

Understanding Core Concepts in Information Retrieval: Lexical and Semantic Gaps, Retrieval Models, and Algorithms

zkyd
Multimodal Semantic Indexing for Image Retrieval at IIIT Hyderabad

Multimodal Semantic Indexing for Image Retrieval at IIIT Hyderabad

kailan
Soil Moisture Retrieval Over Bare Surfaces Using Radar Observations

Soil Moisture Retrieval Over Bare Surfaces Using Radar Observations

ingenito_q
TEMPO Simulation and Retrieval Tools Testing at SAO

TEMPO Simulation and Retrieval Tools Testing at SAO

madgrle
Requirements for Semantic Biobanks and Global Biobank Data Retrieval

Requirements for Semantic Biobanks and Global Biobank Data Retrieval

lei_cen
Efficient Scoring Techniques in Information Retrieval

Efficient Scoring Techniques in Information Retrieval

capelle_j
Assisted Reproductive Techniques: Oocyte Retrieval Process

Assisted Reproductive Techniques: Oocyte Retrieval Process

judson
Comprehensive Overview of Information Retrieval Techniques

Comprehensive Overview of Information Retrieval Techniques

levi_ev
Understanding Evaluation in Education

Understanding Evaluation in Education

imani
Evaluation of FME Zero Emission Neighbourhoods in Smart Cities

Evaluation of FME Zero Emission Neighbourhoods in Smart Cities

raahim
BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

harlem
Understanding Information Retrieval Models and Processes

Understanding Information Retrieval Models and Processes

stallman_a
Efficient Solutions for Large-Scale Text Document Retrieval

Efficient Solutions for Large-Scale Text Document Retrieval

livings

More Related Content