Metaverse-based remote support system with smooth combination of free viewpoint observation and hand gesture instruction

In field operations with large machines, onsite large task spaces have been typically required. In these cases, remote skilled workers have to provide a step-by-step guide by observing the onsite situation around the large task spaces in three dimensions, instructing onsite unskilled workers on how to perform the tasks with the correct hand gestures at the right position, and switching between observation and instruction frequently during the remote support. In this study, we developed a Metaverse-based remote support system with a seamless user interface for switching between free viewpoint observation and hand gesture instruction. The proposed system enables remote skilled workers to observe the onsite field from any viewpoint, transfer hand gesture instruction to onsite workers, and seamlessly switch between free viewpoint observation and free hand gesture instruction without having to change devices. We compared the time efficiency of the proposed system and a conventional system through experiments with 28 users and found that our system improved the observation time efficiency by 65.7%, the instruction time efficiency by 27.9%, and the switching time efficiency between observation and instruction by 14.6%. These results indicate that the proposed system enables remote skilled workers to support onsite workers quickly and efficiently.