Challenges in Virtual Reality Studies: Ethics and Internal and External Validity

User studies on human augmentation nowadays frequently involve virtual reality (VR) technology. This is because VR studies allow augmentations of the human body or senses to be evaluated virtually without having to develop elaborate physical prototypes. However, there are many challenges in VR studies that stem from a multitude of factors. In this paper, we first discuss different types of VR studies and suggest high-level terminology to facilitate further discussions in this space. Then, we derive challenges from the literature that researchers might face when conducting research with VR technology. In particular, we discuss ethics, internal validity, external validity, the technological capabilities of VR hardware, and the costs of VR studies. We further discuss how the challenges might apply to different types of VR studies, and formulate recommendations.