Article,
Escaping the McNamara Fallacy: Toward More Impactful Recommender Systems Research
AI Magazine, 41 (4): 79-95 (December 2020)
DOI: 10.1609/aimag.v41i4.5312
Abstract
Recommender systems are among today’s most successful application areas of AI. However, in the recommender systems research community, we have fallen prey of a McNamara fallacy to a worrying extent: In the majority of our research efforts, we rely almost exclusively on computational measures such as prediction accuracy, which are easier to make than applying other evaluation methods. However, it remains unclear whether small improvements in terms of such computational measures actually matter a lot and whether they lead us to better systems in practice. A paradigm shift in terms of our research culture and goals is therefore needed. We cannot focus exclusively on abstract computational measures any longer, but must direct our attention to research questions that are more relevant and have more impact in the real world. In this work, we review the various ways of how recommender systems may create value; how they, positively or negatively, impact consumers, businesses, and the society; and how we can measure the resulting effects. Through our analyses, we identify a number of research gaps and propose ways of broadening and improving our methodology in a way that leads us to more impactful research in our field.
