Leveraging Out-of-the-Box Retrieval Models to Improve Mental Health Support

Abstract

: This work compares the performance of several information retrieval (IR) models in the search for relevant mental health documents based on relevance to forum post queries from a fully-moderated online mental health service. Three different architectures are assessed: a sparse lexical model, BM25, is used as a base-line, alongside two neural SBERT-based architectures - the bi-encoder and the cross-encoder. We highlight the credibility of using pretrained language models (PLMs) out-of-the-box, without an additional fine-tuning stage, to achieve high retrieval quality across a limited set of resources. Error analysis of the ranking results suggested PLMs make errors on documents which contain so called red-herrings - words which are semantically related but irrelevant to the query - whereas human judgements were found to suffer when queries are vague and present no clear information need. Further, we show that bias towards an author’s writing style within a PLM affects retrieval quality and, therefore, can impact on the success of mental health support if left unaddressed.