DJaytopia: a hybrid intelligent DJ co-remixing system

Abstract

Nowadays, musical mixing platforms are featured with programmed interventions and digitized information visualization to support DJ's performance (Montano 2010), however, the visualization is always obscure to the average music consumers (Beamish, Maclean, and Fels 2004). Being a well-performed DJ requires the level of expertise and experience that most average music consumers lack (Cliff 2000), as every audience has a completely different taste in music (Schäfer and Sedlmeier 2010). This study aims at developing an AI / ML-based system to lower the bar for novice DJs and even average music consumers to create personalized music remixes.Generally, music can be intelligently composed by analyzing harmonic and melodic features to generate genre-specific compositional elements or to alter the compositional structure of a song (Tan and Li 2021). Despite the technical breakthroughs that have been made, listeners have reacted negatively to this music due to the lack of user data to back it up and the neglect of the user's perception of the piece (Tigre Moura and Maw 2021). In a conventional scenario, DJs can express their attitudes towards music preferences by listening to the music directly, which requires a well understanding of the audience's mind. Following the recent launch and explosion of ChatGPT, which has evidenced that an intelligent system could help users innovate by solving their problems in textual form through conversational interactions (Dis et al. 2023; Dwivedi et al. 2023); also collecting the users' feedback through conversations, observing user reactions, and inviting user reviews. Such AI-enabled systems are able to learn about the user's preferred music style and various DJ mixing techniques. This study adopts a typical human-in-the-loop (HITL) approach to develop a crowd-learning music mixing system implementing AI and Virtual Reality technologies. The proposed HITL-based co-music arrangement system should be able to collect musical data and techniques; a VR environment is built to provide users with a platform to record user-created music and corresponding applied methods as well as audience ratings worldwide. After processing the data, users can try out a compilation of songs assisted by a robotic arm. With the help of the robotic arm, it will be easier and faster for users to create collections with a personal touch and more specific techniques. The essential functions include: a) Providing users with an immersive environment to learn the basic operations of the DJ console. b) Collecting the user's preferences for compilation techniques and the content of different DJ's compositions for use through an “immersive online multiplayer music compilation platform” to generate a personalized library of methods to help the user compile songs; c) Assisting the user in creating their preferred individual compilation style faster as they try out the DJ's operations; d) Indicating to the user where the music needs to be equalized, switched or arranged. Instead of showing the user the digital music signal to assist in creating more efficiently, the system directly operates on the DJ console.User experience experiments were conducted with both novice DJs and experienced DJs to validate whether the proposed system could help humans in creating more engaging music with stronger musicality. Five participants, respectively three novice DJs and two experienced DJs, joined two experiments of half an hour on a virtual DJ and an actual DJ console. They started the experiment by experiencing the virtual DJ console and DJ community in VR. They remixed independently first and then collaborated with the robotic arm together for music production on the actual DJ console. Three different audience also joined the experiment to evaluate the performance of users. The result was that the music produced with the robotic arm had better musicality. The user's attitude towards the whole experience, reflected in whether the music was rhythmic or the system was inspiring was recorded in the feedback. Overall, the users had a satisfying and smooth experience, and the collaborative music remixing had a certain level of musicality, but there is still some room for improvement in terms of user understanding. However, the users expressed that this fresh collaborative approach made them more interested in DJing and motivated their desire to learn and create.