Recommendation systems compliant with legal and editorial policies: the BBC+ app journey

Abstract

The BBC produces thousands of pieces of content every day and numerous BBC products deliver this content to millions of users. For many years the content has been manually curated (this is evident in the selection of stories on the front page of the BBC News website and app for example). To support content creation and curation, a set of editorial guidelines have been developed to build quality and trust in the BBC. As personalisation becomes more important for audience engagement, we have been exploring how algorithmically-driven recommendations could be integrated in our products. In this talk we describe how we developed recommendation systems for the BBC+ app that comply with legal and editorial policies and promote the values of the organisation. We also discuss the challenges we face moving forward, extending the use of recommendation systems for a public service media organisation like the BBC. The BBC+ app is the first product to host in-house recommendations in a fully algorithmically-driven application. The app surfaces short video clips and is targeted at younger audiences. The first challenge we dealt with was content metadata. Content metadata are created for different purposes and managed by different teams across the organisation making it difficult to have reliable and consistent information. Metadata enrichment strategies have been applied to identify content that is considered to be editorially sensitive, such as political content, current legal cases, archived news, commercial content, and content unsuitable for an under 16 audience. Metadata enrichment is also applied to identify content that due care has not been taken such as poor titles, and spelling and grammar mistakes. The first versions of recommendation algorithms exclude all editorially risky content from the recommendations, the most serious of which is avoiding contempt of court. In other cases we exclude content that could undermine our quality and trustworthiness. The General Data Protection Regulation (GDPR) that recently came into effect had strong implications on the design of our system architecture, the choice of the recommendation models, and the implementation of specific product features. For example, the user should be able to delete their data or switch off personalisation at any time. Our system architecture should allow us to trace down and delete all data from that user and switch to non-personalised content. The recommendations should also be explainable and this led us to sometimes choosing a simpler model so that it is possible to more easily explain why a user was recommended a particular type of content. Specific product features were also added to enhance transparency and explainability. For example, the user could view their history of watched items, delete any item, and get an explanation of why a piece of content was recommended to them. At the BBC we aim to not only entertain our audiences but also to inform and educate. These BBC values are also reflected in our evaluation strategies and metrics. While we aim to increase audience engagement we are also responsible for providing recent and diverse content that meets the needs of all our audiences. Accuracy metrics such as Hit Rate and Normalized Discounted Cumulative Gain (NDCG) can give a good estimate of the predictive performance of the model. However, recency and diversity metrics have sometimes more weight in our products, especially in applications delivering news content. What is more, qualitative evaluation is very important before releasing any new model into production. We work closely with editorial teams who provide feedback on the quality of the recommendations and flag content not adhering to the BBC's values or the legal and editorial policies. The development of the BBC+ app has been a great journey. We learned a lot about our content metadata, the implications of GDPR in our system, and our evaluation strategies. We created a minimum viable product that is compliant with legal and editorial policies. However, a lot needs to be done to ensure the recommendations meet the quality standards of the BBC. While excluding editorially sensitive content has limited the risk of contempt of court, algorithmic fairness and impartiality still need to be addressed. We encourage the community to look more into these topics and help us create the way forward towards applications with responsible machine learning.