Retrievers (Elser shown here) struggle with later turns and non-standalone questions:

SOTA LLMs struggle with later turns and unanswerable questions:

Sample Conversation:

MTRAG is a challenging benchmark for SOTA LLMs and a great way to evaluate across multiple domains for Retrieval and Generation! MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore synthetic data and LLM-as-a-judge.

Please just message me on slack

Please add me. Thanks!

Please add me!

This is great! Please add me as well!