Imagine that you gave the representatives of the rivalling European countries in the 1700s the knowledge that, in the not-too-distant future, there will be explosive technological growth followed a rapid expansion in the pool of resources that Europe will get access to. Further, you tell them that it’s likely that early progress will be unevenly distributed and it’s quite likely that one country will get a significant strategic advantage.
What would be the effect on their relations and negotiation strategies?
It’s not clear what the actual effect would be, but it seems quite clear how it would affect the optimal strategy:
- Decision-makers in Europe in the 1700s were not used to explosive growth. The expectations were that in the future, international relations will amount to the same old zero-sum competition over a limited set of resources.
- In fact, not too many decades ahead was the industrial revolution which significantly expanded the set of available resources.
- It was also far from obvious which country would be the epicenter of this explosive growth: no one would have predicted that it would be Lancashire, a relative backwater.
- If the expectations of decision-makers were better aligned with the reality that was going to come, it would become relatively clear that a reasonable strategy is to “play it safe” and reach agreements that prevent one party from
Today, we are used to growth on the order of a few percentage points per year, which does not significantly affect calculations within a reasonable decision-making horizon. But if you take seriously the idea that there is a significant chance that AI and other emerging technologies will lead to significantly accelerating growth rates, we might be in a similar (and perhaps more extreme) situation to the one described above.
We have already seen some early signs of pre-commitment to cooperation, such as OpenAI’s capped profit model.
These are the basic ideas behind Eric Drexler’s “Paretotopian argument”: in economic terms, expectations of an expanding Pareto frontier affect optimal strategies for rational agents, assuming diminishing marginal utility with respect to resources. In particular, it makes little sense to pursue overly greedy strategies, as you expose yourself to the risk of foregoing the large benefits of the expanding pie that you would get even with much more cooperative ones.