Digital Yes-Men: How to Deal With Sycophantic Military AI?

Militaries have increasingly embraced decision-support AI for targeting and other planning tasks. An emerging risk identified with respect to these models is ‘sycophancy’: the tendency of AI to align their outputs with their user's views or preferences, even if this view is incorrect. This paper offers an initial perspective on sycophantic AI in the military domain, and identifies the different technical, organisational and operational elements at play to inform more granular research. It examines the phenomenon technically, the risks it introduces to military operations, and the different courses-of-action militaries can take to mitigate this risk. It theorises that sycophancy is militarily deleterious both in the short and long term, by aggravating existing cognitive biases and inducing organisational overtrust, respectively. The paper then explores two main approaches to mitigation that can be taken: technical intervention at the model/design level (e.g., through finetuning), and user training. It theorises that user training is an important complementary measure to technical intervention, since sycophancy can never be comprehensively addressed only at the design stage. Finally, the paper conceptualises tools and procedures militaries could develop to minimise the negative effects sycophantic AI could have on users' decision-making should sycophancy manifest despite all prior efforts at mitigation.