In this CLEF 2014 campaign-style evaluation lab which is based on the News Recommender Systems challenge of ACM RecSys 2013, participants are given the opportunity to develop news recommendation algorithms and have them tested by potentially millions of users in real-time using plista’s Open Recommendation Platform (ORP). The platform receives recommendation requests from various websites offering news articles. Requests are triggered by users visiting those websites.

Participants have access to a virtual machine where they can install their algorithm. The ORP platform forwards the incoming requests to a random virtual machine which produces the recommendation to be delivered to the requester. The random choice is uniformly distributed over all participants. Alternatively, participants may set up their own server to respond to incoming requests. Note that there is a fixed response time limitation. Hence, the participants experience typical restrictions for real­-world recommender systems. Such restrictions pose requirements regarding scalability and computational complexity for the recommendation algorithms. The ORP platform monitors the performance of all participants during the challenge duration by measuring the users’ click-through rate.

Participants have the chance to tune their parameters offline using a provided dataset and can continuously update their parameter settings in order to improve their performance levels. Results will be published on a regular basis to allow participants to compare their performance with respect to baseline and competing approaches.
The performance results will be presented during a half day workshop at CLEF 2014 in Sheffield.

Further Information