IEEE Transactions on Games最新文献

{"title":"Leveraging the OPT Large Language Model for Sentiment Analysis of Game Reviews","authors":"Markos Viggiato;Cor-Paul Bezemer","doi":"10.1109/TG.2023.3313121","DOIUrl":"10.1109/TG.2023.3313121","url":null,"abstract":"Automatically extracting players' sentiments about games can help game developers to better understand the aspects of their games that players like or dislike. Our prior work showed that traditional sentiment analysis techniques do not perform well on game reviews. However, the natural language processing field has seen a steep progress in recent years. In this letter, we follow up on our prior work and investigate how a state-of-the-art large language model (OPT-175B) performs on the sentiment classification of game reviews. We manually analyze the game reviews wrongly classified by OPT-175B to better understand the issues that affect the performance of that model and how those issues compare to the challenges faced by traditional classifiers. We found that OPT-175B achieves (far) better performance than traditional sentiment classifiers, with a 72%-increased \u0000<inline-formula><tex-math>$F$</tex-math></inline-formula>\u0000-measure and a 30%-increased AUC compared to the best traditional classifier studied in our prior work. We also found that common challenges of traditional classifiers, such as reviews with game comparisons and negative terminology, have been mostly solved by the OPT-175B model.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 2","pages":"493-496"},"PeriodicalIF":2.3,"publicationDate":"2023-09-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570261","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MCMARL: Parameterizing Value Function via Mixture of Categorical Distributions for Multi-Agent Reinforcement Learning","authors":"Jian Zhao;Mingyu Yang;Youpeng Zhao;Xunhan Hu;Wengang Zhou;Houqiang Li","doi":"10.1109/TG.2023.3310150","DOIUrl":"10.1109/TG.2023.3310150","url":null,"abstract":"In cooperative multi-agent tasks, a team of agents jointly interact with an environment by taking actions, receiving a team reward, and observing the next state. During the interactions, the uncertainty of environment and reward will inevitably induce stochasticity in the long-term returns, and the randomness can be exacerbated with the increasing number of agents. However, such randomness is ignored by most of the existing value-based multi-agent reinforcement learning (MARL) methods, which only model the expectation of \u0000<inline-formula><tex-math>$Q$</tex-math></inline-formula>\u0000-value for both the individual agents and the team. Compared to using the expectations of the long-term returns, it is preferable to directly model the stochasticity by estimating the returns through distributions. With this motivation, this article proposes a novel value-based MARL framework from a distributional perspective, i.e., parameterizing value function via \u0000<underline>M</u>\u0000ixture of \u0000<underline>C</u>\u0000ategorical distributions for MARL (MCMARL). Specifically, we model both the individual and global \u0000<inline-formula><tex-math>$Q$</tex-math></inline-formula>\u0000-values with categorical distribution. To integrate categorical distributions, we define five basic operations on the distribution, which allow the generalization of expected value function factorization methods (e.g., value decomposition networks (VDN) and QMIX) to their MCMARL variants. We further prove that our MCMARL framework satisfies the \u0000<italic>Distributional-Individual-Global-Max</i>\u0000 principle with respect to the expectation of distribution, which guarantees the consistency between joint and individual greedy action selections in the global and individual \u0000<inline-formula><tex-math>$Q$</tex-math></inline-formula>\u0000-values. Empirically, we evaluate MCMARL on both the stochastic matrix game and the challenging set of \u0000<italic>StarCraft II</i>\u0000 micromanagement tasks, showing the efficacy of our framework.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 3","pages":"556-565"},"PeriodicalIF":1.7,"publicationDate":"2023-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47562284","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deep Reinforcement Learning Using Optimized Monte Carlo Tree Search in EWN","authors":"Yixian Zhang;Zhuoxuan Li;Yiding Cao;Xuan Zhao;Jinde Cao","doi":"10.1109/TG.2023.3308898","DOIUrl":"10.1109/TG.2023.3308898","url":null,"abstract":"<italic>EinStein würfelt nicht!</i>\u0000 (EWN) is a perfect information stochastic game, in which randomness influences the game process enormously. In this article, we propose an optimized algorithm named Quick Neural Network Tree Search (QNNTS) based on deep reinforcement learning and Monte Carlo tree search (MCTS) to construct the artificial intelligence agent of EWN. Meanwhile, the lightness of the model makes it possible to train with much less computing resources. The optimization structure of the algorithm based on MCTS is named Optimized Upper Confidence Bound Applied to Tree with Heuristic Search, which introduces the expectation valuation strategy into the MCTS. As the prerequisite product of QNNTS, it performs with an improvement of the winning rate. Ultimately, the Attention-ResNet structure combined with domain knowledge is used to obtain the proposed algorithm. Compared with several conventional algorithms, it gains high winning rates of at least 68%.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 3","pages":"544-555"},"PeriodicalIF":1.7,"publicationDate":"2023-08-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multigoal Reinforcement Learning via Exploring Entropy-Regularized Successor Matching","authors":"Xiaoyun Feng;Yun Zhou","doi":"10.1109/TG.2023.3304315","DOIUrl":"10.1109/TG.2023.3304315","url":null,"abstract":"Multigoal reinforcement learning (RL) algorithms tend to achieve and generalize over diverse goals. However, unlike single-goal agents, multigoal agents struggle to break through the exploration bottleneck with a fair share of interactions, owing to rarely reusable goal-oriented experiences with sparse goal-reaching rewards. Therefore, well-arranged behavior goals during training are essential for multigoal agents, especially in long-horizon tasks. To this end, we propose efficient multigoal exploration on the basis of maximizing the entropy of successor features and Exploring entropy-regularized successor matching, namely, E\u0000<inline-formula><tex-math>$^{2}$</tex-math></inline-formula>\u0000SM. E\u0000<inline-formula><tex-math>$^{2}$</tex-math></inline-formula>\u0000SM adopts the idea of a successor feature and extends it to entropy-regularized goal-reaching successor mapping that serves as a more stable state feature under sparse rewards. The key contribution of our work is to perform intrinsic goal setting with behavior goals that are more likely to be achieved in terms of future state occupancies as well as promising in expanding the exploration frontier. Experiments on challenging long-horizon manipulation tasks show that E\u0000<inline-formula><tex-math>$^{2}$</tex-math></inline-formula>\u0000SM deals well with sparse rewards and in pursuit of maximal state-covering, E\u0000<inline-formula><tex-math>$^{2}$</tex-math></inline-formula>\u0000SM efficiently identifies valuable behavior goals toward specific goal-reaching by matching the successor mapping.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"15 4","pages":"538-548"},"PeriodicalIF":2.3,"publicationDate":"2023-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Leveraging Joint-Action Embedding in Multiagent Reinforcement Learning for Cooperative Games","authors":"Xingzhou Lou;Junge Zhang;Yali Du;Chao Yu;Zhaofeng He;Kaiqi Huang","doi":"10.1109/TG.2023.3302694","DOIUrl":"10.1109/TG.2023.3302694","url":null,"abstract":"State-of-the-art multiagent policy gradient (MAPG) methods have demonstrated convincing capability in many cooperative games. However, the exponentially growing joint-action space severely challenges the critic's value evaluation and hinders performance of MAPG methods. To address this issue, we augment Central-Q policy gradient with a joint-action embedding function and propose mutual-information maximization MAPG (M3APG). The joint-action embedding function makes joint-actions contain information of state transitions, which will improve the critic's generalization over the joint-action space by allowing it to infer joint-actions' outcomes. We theoretically prove that with a fixed joint-action embedding function, the convergence of M3APG is guaranteed. Experiment results of the \u0000<italic>StarCraft</i>\u0000 multiagent challenge (SMAC) demonstrate that M3APG gives evaluation results with better accuracy and outperform other MAPG basic models across various maps of multiple difficulty levels. We empirically show that our joint-action embedding model can be extended to value-based multiagent reinforcement learning methods and state-of-the-art MAPG methods. Finally, we run an ablation study to show that the usage of mutual information in our method is necessary and effective.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 2","pages":"470-482"},"PeriodicalIF":2.3,"publicationDate":"2023-08-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570196","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Improved Exploration With Demonstrations in Procedurally-Generated Environments","authors":"Mao Xu;Shuzhi Sam Ge;Dongjie Zhao;Qian Zhao","doi":"10.1109/TG.2023.3299986","DOIUrl":"10.1109/TG.2023.3299986","url":null,"abstract":"Exploring sparse reward environments remains a major challenge in model-free deep reinforcement learning (RL). State-of-the-art exploration methods address this challenge by utilizing intrinsic rewards to guide exploration in uncertain environment dynamics or novel states. However, these methods fall short in procedurally-generated environments, where the agent is unlikely to visit a state more than once due to the different environments generated in each episode. Recently, imitation-learning-based exploration methods have been proposed to guide exploration in different kinds of procedurally-generated environments by imitating high-quality exploration episodes. However, these methods have weaker exploration capabilities and lower sample efficiency in complex procedurally-generated environments. Motivated by the fact that demonstrations can guide exploration in sparse reward environments, we propose improved exploration with demonstrations (IEWD), an improved imitation-learning-based exploration method in procedurally-generated environments, which utilizes demonstrations from these environments. IEWD assigns different episode-level exploration scores to each demonstration episode and generated episode. IEWD then ranks these episodes based on their scores and stores highly-scored episodes into a small ranking buffer. IEWD treats these highly-scored episodes as good exploration episodes and makes the deep RL agent imitate exploration behaviors from the ranking buffer to reproduce exploration behaviors from good exploration episodes. Additionally, IEWD adopts the experience replay buffer to store generated positive episodes and demonstrations and employs self-imitating learning to utilize experiences from the experience replay buffer to optimize the policy of the deep RL agent. We evaluate our method IEWD on several procedurally-generated MiniGrid environments and 3-D maze environments from MiniWorld. The results show that IEWD significantly outperforms existing learning from demonstration methods and exploration methods, including state-of-the-art imitation-learning-based exploration methods, in terms of sample efficiency and final performance in complex procedurally-generated environments.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 3","pages":"530-543"},"PeriodicalIF":1.7,"publicationDate":"2023-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Full DouZero+: Improving DouDizhu AI by Opponent Modeling, Coach-Guided Training and Bidding Learning","authors":"Youpeng Zhao;Jian Zhao;Xunhan Hu;Wengang Zhou;Houqiang Li","doi":"10.1109/TG.2023.3299612","DOIUrl":"10.1109/TG.2023.3299612","url":null,"abstract":"With the development of deep reinforcement learning, much progress in various perfect and imperfect information games has been achieved. Among these games, \u0000<italic>DouDizhu</i>\u0000, a popular card game in China, poses great challenges because of the imperfect information, large state and action space as well as the cooperation issue. In this article, we put forward an AI system for this game, which adopts opponent modeling and coach-guided training to help agents make better decisions when playing cards. Besides, we take the bidding phase of \u0000<italic>DouDizhu</i>\u0000 into consideration, which is usually ignored by existing works, and train a bidding network using Monte Carlo simulation. As a result, we achieve a full version of our AI system that is applicable to real-world competitions. We conduct extensive experiments to evaluate the effectiveness of the three techniques adopted in our method and demonstrate the superior performance of our AI over the state-of-the-art \u0000<italic>DouDizhu</i>\u0000 AI, i.e., DouZero. We upload our AI systems, one is bidding-free and the other is equipped with a bidding network, to Botzone platform and they both rank the first among over 400 and 250 AI programs on the two corresponding leaderboards, respectively.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 3","pages":"518-529"},"PeriodicalIF":1.7,"publicationDate":"2023-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570319","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generating Interpretable Play-Style Descriptions Through Deep Unsupervised Clustering of Trajectories","authors":"Branden Ingram;Clint van Alten;Richard Klein;Benjamin Rosman","doi":"10.1109/TG.2023.3299074","DOIUrl":"10.1109/TG.2023.3299074","url":null,"abstract":"In any game, play style is a concept that describes the technique and strategy employed by a player to achieve a goal. Identifying a player's style is desirable as it can enlighten players on which approaches work better or worse in different scenarios and inform developers of the value of design decisions. In previous work, we demonstrated an unsupervised LSTM-autoencoder clustering approach for play-style identification capable of handling multidimensional variable length player trajectories. The efficacy of our model was demonstrated on both complete and partial trajectories in both a simulated and natural environment. Lastly, through state frequency analysis, the properties of each of the play styles were identified and compared. This work expands on this approach by demonstrating a process by which we utilize temporal information to identify the decision boundaries related to particular clusters. Additionally, we demonstrate further robustness by applying the same techniques to \u0000<italic>MiniDungeons</i>\u0000, another popular domain for player modeling research. Finally, we also propose approaches for determining mean play-style examples suitable for describing general play-style behaviors and for determining the correct number of represented play-styles.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"15 4","pages":"507-516"},"PeriodicalIF":2.3,"publicationDate":"2023-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"62570282","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Hierarchically Composing Level Generators for the Creation of Complex Structures","authors":"Michael Beukman;Manuel Fokam;Marcel Kruger;Guy Axelrod;Muhammad Nasir;Branden Ingram;Benjamin Rosman;Steven James","doi":"10.1109/TG.2023.3297619","DOIUrl":"https://doi.org/10.1109/TG.2023.3297619","url":null,"abstract":"Procedural content generation (PCG) is a growing field, with numerous applications in the video game industry and great potential to help create better games at a fraction of the cost of manual creation. However, much of the work in PCG is focused on generating relatively straightforward levels in simple games, as it is challenging to design an optimizable objective function for complex settings. This limits the applicability of PCG to more complex and modern titles, hindering its adoption in the industry. Our work aims to address this limitation by introducing a compositional level generation method that recursively composes simple low-level generators to construct large and complex creations. This approach allows for easily-optimizable objectives and the ability to design a complex structure in an interpretable way by referencing lower-level components. We empirically demonstrate that our method outperforms a noncompositional baseline by more accurately satisfying a designer's functional requirements in several tasks. Finally, we provide a qualitative showcase (in \u0000<italic>Minecraft</i>\u0000) illustrating the large and complex, but still coherent, structures that were generated using simple base generators.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"16 2","pages":"459-469"},"PeriodicalIF":2.3,"publicationDate":"2023-07-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141333979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Mouse Sensitivity in First-Person Targeting Tasks","authors":"Ben Boudaoud;Josef Spjut;Joohwan Kim","doi":"10.1109/TG.2023.3293692","DOIUrl":"https://doi.org/10.1109/TG.2023.3293692","url":null,"abstract":"Mouse sensitivity in first-person targeting tasks is a highly debated issue. Recommendations within a single game can vary by a factor of 10× or more and are an active topic of experimentation in both competitive and recreational esports communities. Inspired by work in pointer-based gain optimization and extending our previous results from the first user study focused on mouse sensitivity in first-person targeting tasks (Boudaoud et al., 2023), we describe a range of optimal mouse sensitivity wherein players perform statistically significantly better in task completion time and throughput. For tasks involving first-person view control, mouse sensitivity is best described using the ratio between an in-game rotation of the view and corresponding physical displacement of the mouse. We discuss how this displacement-to-rotation sensitivity is incompatible with the control-display gain reported in traditional pointer-based gain studies as well as other rotational gains reported in head-controlled interface studies. We provide additional details regarding impacts of mouse dots per inch, on reported sensitivity, the distribution of spatial difficulty in our experiment, our submovement parsing algorithm, and relationships between measured parameters, further demonstrating optimal sensitivity arising from a speed-precision tradeoff. We conclude our work by updating and improving our suggestions for mouse sensitivity selection and refining directions for future work.","PeriodicalId":55977,"journal":{"name":"IEEE Transactions on Games","volume":"15 4","pages":"493-506"},"PeriodicalIF":2.3,"publicationDate":"2023-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138678654","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}