Abstract

Recommender systems add value to vast content resources by matching users with items of interest. In recent years, immense progress has been made in recommendation techniques. The evaluation of these has however not been matched and is threatening to impede the further development of recommender systems. In this paper we propose an approach that addresses this impasse by formulating a novel evaluation concept adopting aspects from recommender systems research and industry. Our model can express the quality of a recommender algorithm from three perspectives, the end consumer (user), the service provider and the vendor (business and technique for both). We review current benchmarking activities and point out their shortcomings, which are addressed by our model. We also explain how our 3D benchmarking framework would apply to a specific use case.

@ARTICLE{Said:2012:RSE,
  author = {Alan Said and Domonkos Tikk and Klara Stumpf and Yue Shi and Martha Larson and Paolo Cremonesi},
  title = {Recommender Systems Evaluation: A 3D Benchmark},
  year = {2012},
  pages = {21--23},
  booktitle = {Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE 2012)},
  location = {Dublin, Ireland},
  series = {RUE'12},
  publisher = {CEUR-WS Vol. 910},
  urn = { urn:nbn:de:0074-910-9 }
}