Recommender Systems Evaluation: A 3D Benchmark


Recommender systems add value to vast content resources by matching users with items of interest. In recent years, immense progress has been made in recommendation techniques. The evaluation of these has however not been matched and is threatening to impede the further development of recommender systems. In this paper we propose an approach that addresses this impasse by formulating a novel evaluation concept adopting aspects from recommender systems research and industry. Our model can express the quality of a recommender algorithm from three perspectives, the end consumer (user), the service provider and the vendor (business and technique for both). We review current benchmarking activities and point out their shortcomings, which are addressed by our model. We also explain how our 3D benchmarking framework would apply to a specific use case.

  author = {Alan Said and Domonkos Tikk and Klara Stumpf and Yue Shi and Martha Larson and Paolo Cremonesi},
  title = {Recommender Systems Evaluation: A 3D Benchmark},
  year = {2012},
  pages = {21--23},
  booktitle = {Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE 2012)},
  location = {Dublin, Ireland},
  series = {RUE'12},
  publisher = {CEUR-WS Vol. 910},
  urn = { urn:nbn:de:0074-910-9 }
Alan Said, Domonkos Tikk, Klara Stumpf, Yue Shi, Martha Larson, Paolo Cremonesi
Conference Paper
ACM RecSys 2012 International Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE '12)