Impact of Iterative Development and Beta-Testing on the Usability and Acceptability of a Novel Just-in-Time Adaptive Digital Physical Activity Intervention.

Abstract

Background: The search for cost-effective population-based physical activity interventions continues. Therefore, we developed a novel just-in-time adaptive digital assistant supported by machine learning (ie, MoveMentor). Beta-testing is essential to evaluate both technical performance and user acceptance. The aim of this study was to assess app usability, acceptability, and technical performance through iterative rounds of beta-testing.

Methods: Insufficiently active people (age: 39.8 [10.2]; 86% female) participated in 2 rounds of beta-testing (round 1, n = 112; round 2, n = 41). Participants downloaded the digital assistant app onto their phone to use during the study period (round 1: 12 wk, round 2: 4 wk). Participants were asked complete at least 4 educational and 5 chat conversations, rate over 50 notifications, and complete an online follow-up survey at week 4 examining aspects of app usability and acceptability. Descriptive statistics and t tests were used to analyze outcomes.

Results: Across both rounds, the app demonstrated good overall usability scores (System Usability Scale: 75.3 out of 100) but lower usefulness ratings. Round 2 participants showed increased engagement with features including action plans (P < .001), educational conversations (P < .001), and personalization features (P < .001), and they appreciated the educational conversations more (P < .05). Technical issues including data syncing problems and chat limitations persisted across both rounds. The notification system received mixed feedback, though customization options in round 2 reduced complaints (12.2%-7.3%).

Conclusions: The app demonstrated good acceptability and usability but low usefulness. The iterative beta-testing successfully identified areas for improvement and enabled meaningful enhancements to content and user engagement features. While some technical challenges persisted, the beta-testing provided clear direction for ongoing improvements.