Abstract

Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development requires a new class of resources. Today's researchers need access to infrastructure enabling multi-dimensional evaluation of recommender systems. In other words, they need to analyze algorithms concerning both functional requirements (such as prediction accuracy) and non-functional requirements (such as speed). Researchers need to subject algorithms to realistic conditions in online A/B tests. We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by "replaying" it locally, and ORP makes it possible to take this evaluation "live" in a living lab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online recommender system resource ever to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in the development of the dataset and ORP.

@inproceedings{Kille:2017:SRM:3077136.3080726,
 author = {Kille, Benjamin and Lommatzsch, Andreas and Hopfgartner, Frank and Larson, Martha and de Vries, Arjen P.},
 title = {A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms},
 booktitle = {Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval},
 series = {SIGIR '17},
 year = {2017},
 isbn = {978-1-4503-5022-8},
 location = {Shinjuku, Tokyo, Japan},
 pages = {1257--1260},
 numpages = {4},
 url = {http://doi.acm.org/10.1145/3077136.3080726},
 doi = {10.1145/3077136.3080726},
 acmid = {3080726},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {multi-dimensional benchmarking, recommender system, streams},
}