Social Environment Design

Eddie Zhang^1,2, Sadie Zhao¹, Tonghan Wang¹, Safwan Hossain¹, Henry Gasztowtt³ Stephan Zheng⁴ David C. Parkes¹ Milind Tambe^1,5 Yiling Chen¹

¹Harvard University ²Humanity Unleashed (Formerly Founding) ³Oxford University ⁴Asari AI ⁵Google Research

ICML 2024

Paper arXiv Code Slides

SED proposes a novel framework for automated policy-making through AI.

Abstract

Artificial Intelligence (AI) holds promise as a technology that can be used to improve government and economic policy-making. This paper proposes a new research agenda towards this end by introducing Social Environment Design, a general framework for the use of AI for automated policy-making that connects with the Reinforcement Learning, EconCS, and Computational Social Choice communities. The framework seeks to capture general economic environments, includes voting on policy objectives, and gives a direction for the systematic analysis of government and economic policy through AI simulation. We highlight key open problems for future research in AI-based policymaking. By solving these challenges, we hope to achieve various social welfare objectives, thereby promoting more ethical and responsible decision making.

Please note that the final two sections in the Beginner-Friendly and Detailed Explanations are identical.

Introduction +

Economic policy formulation is a domain fraught with complexity, with traditional economic models providing limited foresight into the outcomes of policy decisions. Policy-makers must not only understand the immediate implications of individual policies but also their aggregate and long-term effects. In addition, human policy-maker incentives are often not aligned with the interests of the general public, and may instead prioritize special interests or reelection. In light of this, AI-based approaches to policy design that can simulate economies and target different objectives, hold the potential for improved policy understanding and formulation.

Figure 1: The proposed framework. The process begins with voting, where human or AI players report preferences on social welfare objectives to a voting mechanism (1). This defines an objective for the Principal, who designs a parameterized 𝑁-player Partially Observable Markov Game. The players are the same as the voters. This POMG unfolds over several timesteps 𝑇. Following the POMG, game state information is extracted to initiate a new round of voting, with the last POMG state used as the first game state of the new round. This whole process is repeated for 𝜏 timesteps.

In an era where AI is gaining increasing attention in governments, it is timely to understand its potential influence on future policy-making. Ideally, such a framework should satisfy the following desiderata:

Alignment of policy-makers to the values of constituents, whilst ensuring fair and equitable representation.
Sufficient model expressivity to accurately represent the intricate governance structures found in the real world, capturing the subtleties and variances of socio-economic interactions.
Balance expressiveness with with computational tractability, making it feasible to scale to systems with a large number of agents.
Build theoretical understanding, enabling systematic analysis and offering a new lens on complex economic models.

Our work proposes a new framework, Social Environment Design, that lays out an agenda in making progress towards these desiderata. In our framework, illustrated in the figure above, we suggest addressing the concern of a misaligned policy-maker an AI-driven Principal policy-maker who seeks to achieve suggested policy goals, given by a voting mechanism. We capture the complexity of a general economic environment whilst maintaining computational tractability by modeling the economy as a Partially Observable Markov Game (POMG), which maintains a fixed observation space for each agent. Finally, we structure our framework as repeatedly finding Stackelberg Equilbria, enabling theoretical understanding by allowing reduction to simpler subproblems.

Motivating Example: Apple Picking Game +

Figure 2: An example of social environment design as an apple picking game, built with Melting Pot 2.0 (Agapiou et al., 2022). Player agents observe a restricted view of their environment, and receive a mixed reward depending on the apples they collect and the apples their observable neighbors collect. The principal observes both an unrestricted view of the environment and the running totals of all the players’ cumulative extrinsic rewards, where it collects tax and redistributes wealth on the cumulated rewards at the end of every tax period (50 timesteps), similar to Zheng et al. (2022). Rewards for the Principal are determined by the change in value of the objective it is currently assigned by agent votes.

In order to give a motivating example for how preference elicitation for the principal in Social Environment Design can be used to align policy-maker incentives, we have created a Sequential Social Dilemma Game inspired by the Harvest Game proposed in Perolat et al. (2017). The game is illustrated in Figure 2. The aim in Harvest Game is to collect apples, with each apple yielding a reward. If all apples in an area are harvested, they never grow back. The dilemma arises when individual self-interest drives rapid harvesting, which could permanently deplete resources. Thus, agents must sacrifice personal benefit and cooperate for collective well-being. One potential solution to this dilemma is through the use of a central government that taxes and redistributes apples. Thus, we have created a new game inspired by Zheng et al. (2022) in which a principal designs tax rates on apple collection and players vote on Utilitarian (productivity) vs. Egalitarian (equality) objectives for the principal. As players interact within this evolving environment, the principal faces the challenge of crafting policies that balance immediate economic incentives with sustainability goals. In order to achieve this, the principal must foster cooperation among players, guiding them towards the Pareto-Efficient equilibrium that the players have chosen. Preliminary results were promising, but further analysis is warranted.

Where Do We Go From Here? +

Our work proposes a generalized framework that has the potential to improve decision-making — we believe that further work will yield tangibly impactful positive results. However, we recognize that its current level of abstraction may obscure its ultimate potential and application. Our motivating example, while useful as a proof of concept, is clearly distant from real-world economic systems. While we recognize the distance, we posit that both a game where agents harvest apples and the NYSE lie on the same fundamental axis. In fact, we outline two natural axes that separate our example from societally impactful instantiations of SED in the future.

Our axes: Simulation Complexity and Principal Control

The first and more immediately apparent axis is the complexity of the simulation that the Principal is parameterizing. One of the simplest instantiations of the simulation would be two independent agents harvesting apples, each with the desire to have as many apples as possible. In this situation, an optimal parameterization from the Principal might one that discourages overharvesting such that both agents are able to have more apples in the long term. A more complex instantiation, like our motivating example, contains more agents, with rewards based on social interactions, communal welfare, and varying levels of selfishness. Already, this is closer to the real world. As complexity scales, simulations may approximate economics in self-contained villages, then towns, cities, states, and so on.
The second axis represents the control that the Principal has on the simulation. In this case, potential control is directly proportional to ability. While our motivating example is relatively complex, an advanced Principal with a knowledge of game theory and high-level math would be able to fully ‘solve the game’ by parameterizing the simulation with the mathematically optimal tax rate. In a situation with an obscured or dynamic equilibrium, an advanced Principal may have the ability to output adaptive code designed to produce parameterizations. An even more advanced Principal may output neural network weights for a network designed to produce parameterizations. The relationship between control and ability applies to inputs as well as outputs — a more sophisticated Principal would be able to process more sophisticated information from the agents within the simulation. Thus, as the ability of the Principal increases, the potential for a more effective instantiation of Social Environment Design increases as well.

What Are The Steps To Get There? +

Clearly, we have steps to take in two directions — simulating real-world complexity, and designing an incredibly advanced Principal. Really, we have a marathon to run in two directions — this distance represents a colossal gap between the technology that we have and the technology that we need. Though this gap is large, Rome wasn’t built in a day. We foresee many implementations of SED, each better than the last. Here, we present a simplified outline what some of those implementations may look like.

SED v1: Here, we may see a small population in an already-governed community, such as a neighborhood or a mobile home park, that are looking to improve the place that they live. In this hypothetical, each member of the population donates some small sum of money, along with the changes they wish to see — perhaps a new park, or an after-school organization that looks after children before their parents get home from work. The Principal will take into account their desires, allocate the money as it sees fit, and gauge the satisfaction of the population. The Principal will continue to optimize for the satisfaction of the population over repeated monetary allocations, leading to a happier population.

SED v2: In this case, we see a larger and more complex population, and a Principal tasked with more than just monetary allocation. This may take the form of a larger town or city, in which the Principal deals with both monetary allocation, via taxes rather than donations, and the drafting of legal policies. These legal policies may concern the local tax system itself, the operation of public transport and infrastructure, and more — the Principal would have responsibilities akin to the legislature of such a population.

SED v3: As the population and ability of the Principal increases, the possible instantiations for SED grow as well. An advancement from SED v2 may be the utilization of SED within a state in the United States, or for a small small country. Of course, this would be dependent on the success of SED v2, as well as the state of pertinent technology needed for a v3 Principal. This prerequisite applies to each advancement — SED v2 would happen only after the conclusive success of SED v1.

Abstract

Artificial Intelligence, or AI, is an incredibly broad field encompassing topics like Large Language Models (like ChatGPT), machine vision (think of AI looking at X-Rays to find tumors), and much more. Our research focuses on a different implementation, concerning how AI could be applied in government and economics. Specifically, we are proposing a framework in which AI would make more informed and unbiased decisions in government roles. We call this framework Social Environment Design, or SED for short.

Please note that the final two sections in the Beginner-Friendly and Detailed Explanations are identical.

Introduction +

Economic policy is a very complex field, and current economic models are often unable to accurately predict the effects of economic policy decisions. Human policy-makers are tasked with understanding the immediate implication of individual policies, but also their long-term effects and their interactions with other policies. Furthermore, these policy-makers are often subject to bias from lobbyists or their own re-election campaigns. We believe that artificial intelligence represents an opportunity to ensure that economic policy is aligned with the goals of the general population, as well as better managing the information necessary to make these decisions. Thus, our work offers a framework in which AI is in a position to make better informed, unbiased policy decisions.

Figure 1: The proposed framework. In the Inner Simulation, there are several players in an environment. You can imagine each player as a simulated person, and the environment to be some video game. Each player wants to do as well as possible in the game by maximizing their score. After the Inner Simulation has run for some time, the players vote, giving their preferences on the rules of the game. The Principal takes all of these votes, and makes new rules for the game. With the new rules for the game, the Inner Simulation runs until it becomes time to vote again. Then, the process continues.

As an example, let’s picture the Inner Simulation as the game Pac-Man, where Pac-Man and all of the ghosts are the players, and the environment is the maze and all of the pellets. It’s important to note that no actual people are controlling any of the players — Pac-Man moves around on his own, just like the ghosts. Remember that the goal of each player is to get a higher score: Pac-Man gets a higher score by eating more pellets, and the ghosts get higher scores by catching Pac-Man. Pac-Man and the ghosts play in the Inner Simulation for a while, until it becomes time to vote. At that point, they all report their preferences about the Inner Simulation — Pac-Man’s preference might be for more pellets, and ghosts might report their preference to be able to move faster. All of these votes are considered by the Principal — in this example, we can think of the Principal as the person who coded the Pac-Man game. The Principal looks at all of the votes and changes the rules of the Pac-Man game. Then, the new Inner Simulation runs again, and the process continues!
As much as we love Pac-Man, the goal of our work is not to better design video games. However, this framework is generally applicable. Imagine a small town governed by a mayor. After the election, the population of the town would not have guaranteed influence on the actions of the mayor, and instead have to trust that the mayor had their best interests at heart. In this scenario, Social Environment Design would guarantee that the population had a direct say in the usage of their tax dollars, and know that their votes had direct influence in the governance of the town. Beyond eliminating the need for blind trust, the Principal has the advantage of being able to consider much more information, and thus be more contextualized, than any human.

When designing this framework, we wanted to try to satisfy several requirements.

First, the framework should ensure the alignment of the Principal to the values of the players in the Inner Simulation, with fair and equitable representation.
We want to be able to accurately represent the complicated government and economic structures that we see in the real world today.
We need the framework to be efficient — it takes a lot of computer power (expensive!) to run a framework like this, and we want it to be able to scale to large scenarios. You can think of this like fuel-efficiency — we want our miles-per-gallon to be very high.
Lastly, we want this framework to help develop our theoretical understanding of complex economics.

We argue that Social Environment Design does satisfy these requirements! Point 1: In our framework, illustrated in the figure above, we suggest addressing the concern of alignment by using an Principal who seeks to achieve goals that directly stem from votes. The Principal can’t be corrupted by money or re-election, unlike many human decision-makers. Points 2 and 3: Using some fancy math called Game Theory, we are able to create a complex environment, but still remain ‘fuel-efficient’ by hiding most of that environment from each player (just like you! Most of the world is hidden from you right now; you can’t see Antarctica from where you are). Point 4: By structuring our framework as repeated loops of the same process, we can reduce the larger economic problem into many smaller problems, which will enable us to develop better theoretical understanding of complex economics.

In-Depth Example: Harvest Game +

This image shows a scene from Harvest Game, an example that we used to demonstrate Social Environment Design. This example is more complicated than our earlier Pac-Man example, and is closer to a real-world environment.

The 7 players in Harvest Game are in a field, surrounded by groups of apples. Each player is able to harvest apples — but if all of the apples in a group are harvested, they never grow back! Remember that each player is looking to get the highest score. In this example, the score is a bit more complicated than Pac-Man and the ghosts. There are two factors that contribute to the score of a player. The first factor is the amount of apples that the player has harvested, and the second is how many apples the other players have harvested. These factors don’t necessarily have equal weight in deciding the score of a player — it depends on how selfish the player is. Different players have different levels of selfishness. Selfish players care more about how many apples they have, and unselfish players care more about how many apples the other players have.
In Harvest Game dilemma arises when selfishness drives rapid harvesting, which could permanently deplete resources (remember, when all of the apples in an area are harvested, they never grow back). So, the collective well-being of the population is dependent on individual player sacrificing short-term personal benefit. One potential solution to this dilemma is through the use of a central government (or a Principal!) that taxes and redistributes apples. In Harvest Game, players vote on whether they prefer a utilitarian society (valuing overall productivity) or an egalitarian society (valuing equality). As players interact within this evolving environment, the Principal faces the challenge of crafting tax-rates that balance immediate economic incentives with sustainability goals. In order to achieve this, the Principal must foster cooperation among players, guiding them towards the type of society that the players have chosen. The initial results of Harvest Game using the Principal were promising, but further analysis is warranted.

Where Do We Go From Here? +

Our work proposes a generalized framework that has the potential to improve decision-making, and we believe that further work will yield results that are impactful in the real world. However, we understand that our framework is abstract, and so it may be difficult to see its future potential for application; our Harvest Game example, while useful to demonstrate SED, is still clearly distant from real-world economic systems. While we recognize the distance, we argue that Harvest Game lies on the same fundamental axis as the New York Stock Exchange. In fact, we outline two clear axes that separate our example from societally impactful uses of SED in the future.

Our axes: Simulation Complexity and Principal Ability

The first and more immediately apparent axis is the complexity of the simulation (or real world environment). As an example, a simpler instance of Harvest Game would be two selfish players harvesting apples, each having a score that is maximized solely by increasing their own apples. In this situation, the best rules drafted by the Principal might discourage overharvesting of the apples so that both players are able to have more apples in the long term. A more complex instance, like the Harvest Game that we explained, contains more players, with scores decided by individual wealth (apples), communal welfare, and varying levels of selfishness. Already, this is closer to the real world. As complexity scales, simulations may approximate economics in self-contained villages, then towns, cities, states, and so on.
The second axis represents the ability of the Principal to preside over complex environments. In the last ten years, artificial intelligence has advanced exponentially (ChatGPT, as you know it today, was almost unthinkable a decade ago!). However, there is still significant progress to be made. We are still far from the technology that would enable artificial intelligence to play the role of Principal for even a small town. As we continue along this path of advancement, however, the potential Principals will be able to effectively participate in more complex real-world environments.
While our Harvest Game example is fairly complex, an advanced Principal with a knowledge of game theory and high-level math would be able to fully ‘solve the game’ by coming up with the mathematically optimal tax rate. As environments increase in complexity and become more dynamic, there might not be an optimal ‘solution’ that stays the same. A more advanced Principal may have the ability to create adaptive code that can handle these dynamic situations and still produce . This relationship between control and ability applies to inputs as well as outputs — a more sophisticated Principal would be able to process more sophisticated information from the players in the form of voting. This means that decisions from the Prinicpal will be better contextualized and informed. So, as the ability of the Principal increases, it is able to make better rules for more complex scenarios.

What Are The Steps To Get There? +

SED v1: Here, we may see a small population in an already-governed community, such as a neighborhood or a mobile home park, that are looking to improve the place that they live. In this hypothetical, each member of the population donates some small sum of money, and provide with the changes they wish to see — perhaps a new park, or an after-school organization that looks after children before their parents get home from work. The Principal will take into account their desires, allocate the money as it sees fit, and gauge the satisfaction of the population. The Principal will continue to optimize for the satisfaction of the population over repeated monetary allocations, leading to a happier population.

Brief Summary

Human policy-makers have many issues, such as prioritizing lobbyist interests over the public [1] and corruption [2].
AI, if made absolutely safe and extremely intelligent, could someday do much better. We call this safe AGI.
SED is a model for using safe AGI to enable better policy-making.
It works by democratically gathering values [4,5], creating automated policy [6], and finally simulating policy outcomes [7,8].

Related Work

Social Environment Design would not be possible without some exceptional prior work.
Firstly, Human Centered Mechanism Design with Democratic AI inspired us to solve the problem of alignment through human feedback, designing policies via RLHF. Furthermore, The AI Economist provided strong evidence of the ability of AI to outperform other approaches in complex economic policy design. Together, these works build the foundation on which we propose Social Environment Design.

Lastly, we give thanks to Melting Pot, which allowed us to build our motivating example, inspired by Harvest Game.
The full list of related work can be found in our paper.

BibTeX

@inproceedings{zhang2024sed,
title={Social Environment Design},
author={Edwin Zhang and Sadie Zhao and Tonghan Wang and Safwan Hossain and Henry Gasztowtt and Stephan Zheng and David C. Parkes and Milind Tambe and Yiling Chen},
booktitle={International Conference on Machine Learning},
year={2024},
url={https://arxiv.org/abs/2402.14090}
}