Abstract

Traditionally, the evaluation of recommender algorithms focuses on the offline evaluation using static, set-based datasets. For many web-based application scenarios the "static" evaluation does fit the characteristics of the scenario. The News Recommendation Lab (NewsREEL) challenge wants to encourage researchers to focus on the more realistic online evaluation of recommender algorithms. The NewsREEL challenge allows researchers to evaluate algorithms for recommending online news articles. The algorithms are analyzed both online and offline taking into account the recommendation precision as well as technical aspects. The talk explains the NewsREEL challenge in Detail. We discuss the addressed challenges, review the implemented approaches, and explain our experiences gained organizing the challenge.