Publications Details

Publication Details

Argument Search

Assessing Argument Relevance

authored by
Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenkötter, Nico Weidmann, Henning Wachsmuth, Benno Stein, Matthias Hagen

We report on the first user study on assessing argument relevance. Based on a search among more than 300,000 arguments, four standard retrieval models are compared on 40 topics for 20 controversial issues: every issue has one topic with a biased stance and another neutral one. Following TREC, the top results of the different models on a topic were pooled and relevance-judged by one assessor per topic. The assessors also judged the arguments' rhetorical, logical, and dialectical quality, the results of which were cross-referenced with the relevance judgments. Furthermore, the assessors were asked for their personal opinion, and whether it matched the predefined stance of a topic. Among other results, we find that Terrier's implementations of DirichletLM and DPH are on par, significantly outperforming TFIDF and BM25. The judgments of relevance and quality hardly correlate, giving rise to a more diverse set of ranking criteria than relevance alone. We did not measure a significant bias of assessors when their stance is at odds with a topic's stance.

External Organisation(s)
Leipzig University
University of Stuttgart
University of Bremen
Karlsruhe Institute of Technology (KIT)
Paderborn University
Bauhaus-Universität Weimar
Martin Luther University Halle-Wittenberg
Conference contribution
No. of pages
Publication date
Publication status
Peer reviewed
ASJC Scopus subject areas
Information Systems, Applied Mathematics, Software
Electronic version(s) (Access: Closed)