Modern French Poetry Generation with RoBERTa and GPT-2

Abstract

We present a novel neural model for modern poetry gen- eration in French. The model consists of two pretrained neural models that are fine-tuned for the poem gener- ation task. The encoder of the model is a RoBERTa based one while the decoder is based on GPT-2. This way the model can benefit from the superior natural language understanding performance of RoBERTa and the good natural language generation performance of GPT-2. Our evaluation shows that the model can cre- ate French poetry successfully. On a 5 point scale, the lowest score of 3.57 was given by human judges to typ- icality and emotionality of the output poetry while the best score of 3.79 was given to understandability .