Working papers

Can an AI agent hit a moving target?

Current version: October 2022. 

Abstract:

I model the belief formation and decision making processes of economic agents during a monetary policy regime change (an acceleration in the money supply) with a deep reinforcement learning algorithm in the AI literature. I show that when the money supply accelerates, the learning agents only adjust their actions, which include consumption and demand for real balance, after gathering learning experience for many periods. This delayed adjustments leads to low returns during transition periods. Once they start adjusting to the new environment, their welfare improves. Their changes in beliefs and actions lead to temporary inflation volatility. I also show that, 1. the AI agents who explores their environment more adapt to the policy regime change quicker, which leads to welfare improvements and less inflation volatility, and 2. the AI agents who have experienced a structural change adjust their beliefs and behaviours quicker than an inexperienced learning agent.

AI and Macroeconomics Modelling: Deep Reinforcement Learning in an RBC Model

with Tohid Atashbar 

Current version: IMF Working Paper No. 2023/040 February 2023

Abstract:

This study seeks to construct a basic reinforcement learning-based AI-macroeconomic simulator. We use a deep RL (DRL) approach (DDPG) in an RBC macroeconomic model. We set up two learning scenarios, one of which is deterministic without the technological shock and the other is stochastic. The objective of the deterministic environment is to compare the learning agent's behavior to a deterministic steady-state scenario. We demonstrate that in both deterministic and stochastic scenarios, the agent's choices are close to their optimal value. We also present cases of unstable learning behaviours. This AI-macro model may be enhanced in future research by adding additional variables or sectors to the model or by incorporating different DRL algorithms.

Deep Reinforcement Learning: Emerging Trends in Macroeconomics and Future Prospects

with Tohid Atashbar 

Current version: IMF Working Paper No. 2022/259 December 2022

Abstract:

The application of Deep Reinforcement Learning (DRL) in economics has been an area of active research in recent years. A number of recent works have shown how deep reinforcement learning can be used to study a variety of economic problems, including optimal policy-making, game theory, and bounded rationality. In this paper, after a theoretical introduction to deep reinforcement learning and various DRL algorithms, we provide an overview of the literature on deep reinforcement learning in economics, with a focus on the main applications of deep reinforcement learning in macromodeling. Then, we analyze the potentials and limitations of deep reinforcement learning in macroeconomics and identify a number of issues that need to be addressed in order for deep reinforcement learning to be more widely used in macro modeling.

Learning from Zero: How to Make Consumption-Saving Decisions in a Stochastic Environment with an AI Algorithm

Current version: August 2022

Previous version: CESifo Working Paper version: No. 9255 (August 2021)

Abstract:

I model the learning behaviours of a representative household in a stochastic optimal growth environment. I model explicitly this learning agent's decision-making strategy, their subjective belief, and their memory (and how the memory is used to update beliefs). This agent makes exploratory consumption-saving decisions facing income shocks. I impose a temporary shock and a permanent change to the agent's income process. Their behaviours facing changes are adaptive and follow the qualitative predictions of existing economic theories. Depends on how exploratory the agent is, their quantitative behaviours are different and lead to heterogeneous welfare. I also compare behaviours between AI agents and a rational expectations agent. 

Deep Reinforcement Learning in a Monetary Model

joint with Mingli Chen, Andreas Joseph, Michael Kumhof, Xinlei Pan,  and Xuan Zhou.

April 2021. Rebuilding Macroeconomics Working Paper Series No.58