MTEB logo MTEB
retrieval text

LoTTE

Cite this task

@inproceedings{santhanam-etal-2022-colbertv2,
  address = {Seattle, United States},
  author = {Santhanam, Keshav  and
Khattab, Omar  and
Saad-Falcon, Jon  and
Potts, Christopher  and
Zaharia, Matei},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  doi = {10.18653/v1/2022.naacl-main.272},
  editor = {Carpuat, Marine  and
de Marneffe, Marie-Catherine  and
Meza Ruiz, Ivan Vladimir},
  month = jul,
  pages = {3715--3734},
  publisher = {Association for Computational Linguistics},
  title = {{C}ol{BERT}v2: Effective and Efficient Retrieval via Lightweight Late Interaction},
  url = {https://aclanthology.org/2022.naacl-main.272/},
  year = {2022},
}

LoTTE (Long-Tail Topic-stratified Evaluation for IR) is designed to evaluate retrieval models on underrepresented, long-tail topics. Unlike MSMARCO or BEIR, LoTTE features domain-specific queries and passages from StackExchange (covering writing, recreation, science, technology, and lifestyle), providing a challenging out-of-domain generalization benchmark.

Domains: AcademicWebSocial
Languages: English
Task type
Retrieval
Main metric
hit_rate_at_5
Source dataset
mteb/LoTTE
License
mit
Dates
2021-12-02 → 2022-06-10
Annotations
derived
Sample creation
found
Models scored …
Dataset preview mteb/LoTTE Open on HuggingFace →

Model scores

Loading…