Christopher J. Hanna, R. Hickey, D. Charles, Michaela M. Black
{"title":"Modular Reinforcement Learning architectures for artificially intelligent agents in complex game environments","authors":"Christopher J. Hanna, R. Hickey, D. Charles, Michaela M. Black","doi":"10.1109/ITW.2010.5593329","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593329","url":null,"abstract":"Recently there has been much research focus on the use of Reinforcement Learning (RL) algorithms for game agent control. However, although it has been shown that such agents are capable of learning in real time, the high dimensionality of agent sensor state spaces still prove to be a significant barrier to progress. This paper outlines an approach to dealing with this issue by using a modular RL architecture with a fine granularity of modules. The modular approach enables a reduction of the dimensionality in complex game-like environments by dividing the state space into smaller, more manageable sub tasks. While this approach is successful in reducing dimensionality, challenges with action selection, exploration and reward allocation arise. This paper discusses approaches to overcoming these issues.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115402377","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Partial observability during predictions of the opponent's movements in an RTS game","authors":"S. Butler, Y. Demiris","doi":"10.1109/ITW.2010.5593374","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593374","url":null,"abstract":"In RTS-style games it is important to be able to predict the movements of the opponent's forces to have the best chance of performing appropriate counter-moves. Resorting to using perfect global state information is generally considered to be ‘cheating’ by the player, so to perform such predictions scouts (or observers) must be used to gather information. This means being in the right place at the right time to observe the opponent. In this paper we show the effect of imposing partial observability onto an RTS game with regard to making predictions, and we compare two different mechanisms that decide where best to direct the attention of the observers to maximise the benefit of predictions.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116642494","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Introducing time in emotional behavior networks","authors":"Anja Johansson, Pierangelo Dell'Acqua","doi":"10.1109/ITW.2010.5593342","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593342","url":null,"abstract":"In realistic scenarios an agent should not assume that the effects of its action are immediate. However, behavior networks have not previously been designed to model this, but rather have assumed that all effects are immediate. In this paper we introduce effect delay time and time-discounting into the decision making module of our agent architecture. We describe the previous emotional behavior network, introduce the concept of effect delay and let emotions influence how much time-discounting should be made to the delay time. Finally, we demonstrate the strength of our model by simulating two decision making problems.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116482355","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
B. Weber, Peter A. Mawhorter, Michael Mateas, A. Jhala
{"title":"Reactive planning idioms for multi-scale game AI","authors":"B. Weber, Peter A. Mawhorter, Michael Mateas, A. Jhala","doi":"10.1109/ITW.2010.5593363","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593363","url":null,"abstract":"Many modern games provide environments in which agents perform decision making at several levels of granularity. In the domain of real-time strategy games, an effective agent must make high-level strategic decisions while simultaneously controlling individual units in battle. We advocate reactive planning as a powerful technique for building multi-scale game AI and demonstrate that it enables the specification of complex, real-time agents in a unified agent architecture. We present several idioms used to enable authoring of an agent that concurrently pursues strategic and tactical goals, and an agent for playing the real-time strategy game StarCraft that uses these design patterns.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126332498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Analyzing the Evolution of Social Groups in World of Warcraft®","authors":"Christian Thurau, C. Bauckhage","doi":"10.1109/ITW.2010.5593358","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593358","url":null,"abstract":"This paper investigates the evolution of social structures in the game WORLD OF WARCRAFT®. We analyze 192 million recordings of 18 million characters belonging to 1.4 million teams, spanning a period of 4 years. Using a recent matrix factorization method, we extract lower dimensional data embeddings. The embeddings provide intuitively interpretable categorizations and we find a tendency towards guilds comprised of casual gamers. To our knowledge, this is the first study considering such a vast amount of data for analyzing groups in MMORPGs.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131379033","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Florent Levillain, J. Orero, M. Rifqi, B. Bouchon-Meunier
{"title":"Characterizing player's experience from physiological signals using fuzzy decision trees","authors":"Florent Levillain, J. Orero, M. Rifqi, B. Bouchon-Meunier","doi":"10.1109/ITW.2010.5593370","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593370","url":null,"abstract":"In the recent years video games have enjoyed a dramatic increase in popularity, the growing market being echoed by a genuine interest in the academic field. With this flourishing technological and theoretical efforts, there is need to develop new evaluative methodologies for acknowledging the various aspects of the player's subjective experience, and especially the emotional aspect. In this study, we addressed the possibility of developing a model for assessing the player's enjoyment (amusement) with respect to challenge in an action game. Our aim was to explore the viability of a generic model for assessing emotional experience during gameplay from physiological signals. In particular, we propose an approach to characterize the player's subjective experience in different psychological levels of enjoyment from physiological signals using fuzzy decision trees.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"193 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134305465","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Overtaking opponents with blocking strategies using fuzzy logic","authors":"E. Onieva, L. Cardamone, D. Loiacono, P. Lanzi","doi":"10.1109/ITW.2010.5593364","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593364","url":null,"abstract":"In car racing, blocking refers to maneuvers that can prevent, disturb or possibly block an overtaking action by an incoming car. In this paper, we present an advanced overtaking behavior that is able to deal with opponents implementing advanced blocking strategies. The behavior we developed has been integrated in an existing fuzzy-based architecture for driving simulated cars and tested using The Open Car Racing Simulator (TORCS). We compared a driver implementing our overtaking strategy against six of the bots available in the TORCS distribution and simplix, a state-of-the-art bot which won the 2009 TORCS Endurance World Championship. The comparison was carried out against opponents implementing three blocking strategies of increasing difficulty. The results we present show that our strategy can overtake the opponent car in all the considered scenarios. In contrast, all the other bots can complete the overtaking maneuvers in only less than 40% of the cases. Our strategy is slightly more risky than others and may result in limited rear and lateral damage. Other more cautious drivers receive almost no damage, however they can overtake only around 30% of the cases.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131833478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jan Quadflieg, M. Preuss, Oliver Kramer, G. Rudolph
{"title":"Learning the track and planning ahead in a car racing controller","authors":"Jan Quadflieg, M. Preuss, Oliver Kramer, G. Rudolph","doi":"10.1109/ITW.2010.5593327","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593327","url":null,"abstract":"We propose a robust approach for learning car racing track models from sensory data for the car racing simulator TORCS. Our track recognition system is based on the combination of an advanced preprocessing step of the sensory data and a simple classifier that delivers six types of track shapes similar to the ones a human would recognize. Out of these, establishing a complete track model is straightforward. This model provides an information advantage to controller strategies, as it generally enables planning. We demonstrate how such a planning controller can be derived by a mixture of expert knowledge and a simple evolutionary learning approach and give experimental evidence that knowing not only the current conditions but also the big picture of the track is beneficial, as may be expected.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"45 5","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"113941986","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
N. Bell, Xinghong Fang, Rory Hughes, G. Kendall, Edward O'Reilly, Shenghui Qiu
{"title":"Ghost direction detection and other innovations for Ms. Pac-Man","authors":"N. Bell, Xinghong Fang, Rory Hughes, G. Kendall, Edward O'Reilly, Shenghui Qiu","doi":"10.1109/ITW.2010.5593320","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593320","url":null,"abstract":"Ms. Pac-Man was developed in the 1980s, becoming one of the most popular arcade games of its time. It still has a significant following today and has recently attracted the attention of artificial intelligence researchers, in part, due to the fact that the agent must react in real time in order to navigate its way through the maze.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131210445","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
É. Beaudry, Francis Bisson, S. Chamberland, F. Kabanza
{"title":"Using Markov decision theory to provide a fair challenge in a roll-and-move board game","authors":"É. Beaudry, Francis Bisson, S. Chamberland, F. Kabanza","doi":"10.1109/ITW.2010.5593380","DOIUrl":"https://doi.org/10.1109/ITW.2010.5593380","url":null,"abstract":"Board games are often taken as examples to teach decision-making algorithms in artificial intelligence (AI). These algorithms are generally presented with a strong focus on winning the game. Unfortunately, a few important aspects, such as the gaming experience of human players, are often missing from the equation. This paper presents a simple board game we use in an introductory course in AI to initiate students to the gaming experience issue. The Snakes and Ladders game has been modified to provide different levels of challenges for students. The game with such modifications offers theoretical, algorithmic and programming challenges. One of the most complex is the generation of an optimal policy to provide a fair challenge to an opponent. A solution based on Markov Decision Processes (MDPs) is presented. This approach relies on a simple model of the opponent's playing behaviour.","PeriodicalId":394649,"journal":{"name":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131918480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}