{"title":"通过强化学习和代理建模模拟理性对经济的影响","authors":"Simone Brusatin, Tommaso Padoan, Andrea Coletta, Domenico Delli Gatti, Aldo Glielmo","doi":"arxiv-2405.02161","DOIUrl":null,"url":null,"abstract":"Agent-based models (ABMs) are simulation models used in economics to overcome\nsome of the limitations of traditional frameworks based on general equilibrium\nassumptions. However, agents within an ABM follow predetermined, not fully\nrational, behavioural rules which can be cumbersome to design and difficult to\njustify. Here we leverage multi-agent reinforcement learning (RL) to expand the\ncapabilities of ABMs with the introduction of fully rational agents that learn\ntheir policy by interacting with the environment and maximising a reward\nfunction. Specifically, we propose a 'Rational macro ABM' (R-MABM) framework by\nextending a paradigmatic macro ABM from the economic literature. We show that\ngradually substituting ABM firms in the model with RL agents, trained to\nmaximise profits, allows for a thorough study of the impact of rationality on\nthe economy. We find that RL agents spontaneously learn three distinct\nstrategies for maximising profits, with the optimal strategy depending on the\nlevel of market competition and rationality. We also find that RL agents with\nindependent policies, and without the ability to communicate with each other,\nspontaneously learn to segregate into different strategic groups, thus\nincreasing market power and overall profits. Finally, we find that a higher\ndegree of rationality in the economy always improves the macroeconomic\nenvironment as measured by total output, depending on the specific rational\npolicy, this can come at the cost of higher instability. Our R-MABM framework\nis general, it allows for stable multi-agent learning, and represents a\nprincipled and robust direction to extend existing economic simulators.","PeriodicalId":501487,"journal":{"name":"arXiv - QuantFin - Economics","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-05-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Simulating the economic impact of rationality through reinforcement learning and agent-based modelling\",\"authors\":\"Simone Brusatin, Tommaso Padoan, Andrea Coletta, Domenico Delli Gatti, Aldo Glielmo\",\"doi\":\"arxiv-2405.02161\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Agent-based models (ABMs) are simulation models used in economics to overcome\\nsome of the limitations of traditional frameworks based on general equilibrium\\nassumptions. However, agents within an ABM follow predetermined, not fully\\nrational, behavioural rules which can be cumbersome to design and difficult to\\njustify. Here we leverage multi-agent reinforcement learning (RL) to expand the\\ncapabilities of ABMs with the introduction of fully rational agents that learn\\ntheir policy by interacting with the environment and maximising a reward\\nfunction. Specifically, we propose a 'Rational macro ABM' (R-MABM) framework by\\nextending a paradigmatic macro ABM from the economic literature. We show that\\ngradually substituting ABM firms in the model with RL agents, trained to\\nmaximise profits, allows for a thorough study of the impact of rationality on\\nthe economy. We find that RL agents spontaneously learn three distinct\\nstrategies for maximising profits, with the optimal strategy depending on the\\nlevel of market competition and rationality. We also find that RL agents with\\nindependent policies, and without the ability to communicate with each other,\\nspontaneously learn to segregate into different strategic groups, thus\\nincreasing market power and overall profits. Finally, we find that a higher\\ndegree of rationality in the economy always improves the macroeconomic\\nenvironment as measured by total output, depending on the specific rational\\npolicy, this can come at the cost of higher instability. Our R-MABM framework\\nis general, it allows for stable multi-agent learning, and represents a\\nprincipled and robust direction to extend existing economic simulators.\",\"PeriodicalId\":501487,\"journal\":{\"name\":\"arXiv - QuantFin - Economics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-05-03\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - QuantFin - Economics\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2405.02161\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - QuantFin - Economics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2405.02161","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Simulating the economic impact of rationality through reinforcement learning and agent-based modelling
Agent-based models (ABMs) are simulation models used in economics to overcome
some of the limitations of traditional frameworks based on general equilibrium
assumptions. However, agents within an ABM follow predetermined, not fully
rational, behavioural rules which can be cumbersome to design and difficult to
justify. Here we leverage multi-agent reinforcement learning (RL) to expand the
capabilities of ABMs with the introduction of fully rational agents that learn
their policy by interacting with the environment and maximising a reward
function. Specifically, we propose a 'Rational macro ABM' (R-MABM) framework by
extending a paradigmatic macro ABM from the economic literature. We show that
gradually substituting ABM firms in the model with RL agents, trained to
maximise profits, allows for a thorough study of the impact of rationality on
the economy. We find that RL agents spontaneously learn three distinct
strategies for maximising profits, with the optimal strategy depending on the
level of market competition and rationality. We also find that RL agents with
independent policies, and without the ability to communicate with each other,
spontaneously learn to segregate into different strategic groups, thus
increasing market power and overall profits. Finally, we find that a higher
degree of rationality in the economy always improves the macroeconomic
environment as measured by total output, depending on the specific rational
policy, this can come at the cost of higher instability. Our R-MABM framework
is general, it allows for stable multi-agent learning, and represents a
principled and robust direction to extend existing economic simulators.