Diophantine cryptography in free metabelian groups: Theoretical base

Abstract In this paper we study so-called Diophantine cryptology, a collection of cryptographic schemes where the computational security assumptions are based on hardness of solving some Diophantine equations, and some general ideas and techniques that occur in this area. In particular, we study an interesting variation of the endomorphism problem in groups, termed the double endomorphism problem. We prove that this problem is undecidable in free metabelian groups of sufficiently large rank. We relate this result to computational security assumptions of some group-based cryptosystems. In particular, we show how to improve the Grigoriev–Shpilrain's protocol to get a new computational security assumption based on the double endomorphism problem, providing a better theoretical foundation to security.