Argument Search

Assessing Argument Relevance

authored by

Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenkötter, Nico Weidmann, Henning Wachsmuth, Benno Stein, Matthias Hagen

Abstract

We report on the first user study on assessing argument relevance. Based on a search among more than 300,000 arguments, four standard retrieval models are compared on 40 topics for 20 controversial issues: every issue has one topic with a biased stance and another neutral one. Following TREC, the top results of the different models on a topic were pooled and relevance-judged by one assessor per topic. The assessors also judged the arguments' rhetorical, logical, and dialectical quality, the results of which were cross-referenced with the relevance judgments. Furthermore, the assessors were asked for their personal opinion, and whether it matched the predefined stance of a topic. Among other results, we find that Terrier's implementations of DirichletLM and DPH are on par, significantly outperforming TFIDF and BM25. The judgments of relevance and quality hardly correlate, giving rise to a more diverse set of ranking criteria than relevance alone. We did not measure a significant bias of assessors when their stance is at odds with a topic's stance.

External Organisation(s)

Leipzig University
University of Stuttgart
University of Bremen
Karlsruhe Institute of Technology (KIT)
Paderborn University
Bauhaus-Universität Weimar
Martin Luther University Halle-Wittenberg

Type

Conference contribution

Pages

1117-1120

No. of pages

4

Publication date

18.07.2019

Publication status

Published

Peer reviewed

Yes

ASJC Scopus subject areas

Information Systems, Applied Mathematics, Software

Electronic version(s)

https://doi.org/10.1145/3331184.3331327 (Access: Closed)