Skip to main content
W&B Fully Connected
>
Articles
>
OpenAI

An Interview With Miles Brundage From OpenAI

In this article, we take a look into Miles Brundage's work on the responsible development and impact of AI
Krisha Samir Mehta
Created on September 18|Last edited on November 6
Miles Brundage is a Research Scientist on the Policy team at OpenAI where he studies the societal impacts of artificial intelligence and how to make sure they go well. His research includes responsible innovation, development, and governance of AI. He also focuses on the beneficial and malicious ways in which AI can be used. One of his most significant works is helping devise release strategies for the popular GPT-2 model.
Our goal in this article is to showcase his work throughout these years, and learn more about his research, and the impact it has on the AI community.



Table of Contents

Table of Contents1. Staged Release2. Partnerships3. EngagementsThe Benefits and Costs of Responsible AI DevelopmentThe Need for Collective Action on Responsible AI DevelopmentStrategies to Improve AI Industry Cooperation on SafetyConclusion



Release Strategies and the Social Impacts of Language Models

In February 2019, OpenAI took the world of NLP by storm when they released GPT-2, a large-scale language model used to generate paragraphs of texts. GPT-2 came with a broad range of benefits. The model can provide grammar assistance, generate poetry, and can be used to autocomplete both code and sentences, among other uses. For the model to be so versatile and its applications so diverse, it was trained on a massive dataset of eight million web pages.
The samples on their website demonstrate how well the model performs (My personal favorite was to read about the four-horned silver-white unicorns). However, such a model can also be used for malicious purposes. These include the generation of fake news, impersonating people online, and producing abusive content on social media. Understanding the risks of releasing such a model, OpenAI adopted various measures to ensure that GPT-2 does not have negative social impacts. Let us look at the three main measures taken.

1. Staged Release

OpenAI developed four variants of GPT-2 with the smallest one consisting of 124 million parameters and the largest one consisting of ~1.5 billion parameters. It is a well-known fact in the field of artificial intelligence that bigger models tend to have better performance. This means that though the largest variant has better performance and more benefits than the smaller variant, it can also be misused more.
To avoid any potential misuse, OpenAI came up with a staged release process. The timeline of these releases is mentioned below:
  • February 2019: 124 million parameters model
  • May 2019: 355 million parameters model
  • August 2019: 774 million parameters model
  • November 2019: 1.5 billion parameters model
  • Following this, various other universities and companies adopted such a release strategy.

2. Partnerships

OpenAI partnered with four leading organizations to study GPT-2. Below is an excerpt from the paper describing these partnerships:
"We are excited to be partnering with the following organizations to study GPT-2:
  • Cornell University is studying human susceptibility to digital disinformation generated by language models.
  • The Middlebury Institute of International Studies Center on Terrorism, Extremism, and Counterterrorism (CTEC) is exploring how GPT-2 could be misused by terrorists and extremists online.
  • The University of Oregon is developing a series of “bias probes” to analyze bias within GPT-2.
  • The University of Texas at Austin is studying the statistical detectability of GPT-2 outputs after fine-tuning the model on domain-specific datasets, as well as the extent of detection transfer across different language models."
One of the most influential outcomes of these partnerships was the publication of a generic version of the legal agreement the organizations signed to carry out this study. Such an agreement will become extremely important in the coming years where partnerships will be critical to ensure responsible AI development.

3. Engagements

OpenAI has discussed the impact of GPT-2 with various members of the AI community, researchers, and activists. They also spoke about GPT-2 and their strategy to release it in phases at the AI for Social Good workshop at ICLR. Along with this, they contributed to the Partnership on AI.
OpenAI did not see any significant action towards the misuse of the smaller versions of GPT-2. This provided them with the confidence to release the biggest GPT-2 model in November 2019. While proactive monitoring will still be necessary to ensure that the models are used ethically, GPT-2 opens up a whole new world of AI applications.

The Role of Cooperation in Responsible AI Development

Taking a step back from GPT-2, let us now look at the bigger picture, that of private AI development, and how to ensure that it is done responsibly.

What is responsible AI development?

Brundage also contributed to a paper (led by Amanda Askell) which defines responsible AI development as taking steps to ensure that AI systems have an acceptably low risk of harming their users or society and, ideally, to increase their likelihood of being socially beneficial.
Responsible AI development is an end-to-end process. It ensures that the system is developed is tested for safety and security during development, evaluating potential risks and social impacts before release, and be willing to either change or abandon the project in cases where it poses a risk to people.
Accepting and adopting responsible AI development will come at some cost to companies and organizations.

The Benefits and Costs of Responsible AI Development

It is no surprise that ensuring responsible AI development will come at a cost to companies. This cost can be in the form of investing more time and money to ensure the systems are safe and secure.
The authors explain the concept of "front-runners," where a company making the first development receives significant benefits. This is true if the technology can be patented. However, this front-runner advantage does not always exist. The advantage is not very high in a field like AI where there is no definitive end goal. Companies are in a constant race to develop state-of-the-art technology. The small gap in progress among these companies does not help either.
If a first-mover advantage in AI is weak or non-existent then companies are less likely to engage in a race to the bottom on safety since speed is of lower value. Instead of offering predictions, this paper should be thought of as an analysis of more pessimistic scenarios that involve at least a moderate first mover advantage.
However, there are benefits that a company can reap if it adopts responsible AI development. Product safety through these years is largely driven by market forces. When this safety is compromised, it can lead to a loss of reputation which can be very expensive. Liability laws and regulations further help ensure that the products are safe for use.
So the question we should be asking is, are existing incentives for responsible AI enough?
The authors answer this question in five parts. While we mention them briefly in this report, they are explained in much greater depth in the paper.
The authors believe that product safety primarily comes from market forces, liability laws and industry or government regulations. However, these may not be enough to guarantee responsible AI development due to five main reasons.
  1. Given the increasing complexity of AI systems, it is difficult for consumers to evaluate how safe these systems are. That fact that it is difficult to explain how a neural network makes a prediction only adds more complications.
  2. Since it is difficult to interpret machine learning models, it is difficult to predict why they fail. Hence, even regulators find it tough to assess how safe or dangerous a product is.
  3. The authors explain that harm caused by AI products is likely to affect third parties. Harms from AI systems are difficult to internalize. Along with this, for long-term effects and harms, no one company can be held accountable.
  4. Currently, there is a lack of AI regulation by the government or the industry. It is tricky to get it right since it needs the regulators to have a deep understanding of AI.
  5. Some people have predicted that development in AI will be discontinuous. This means that after a phase of rapid progress, a company may make a large leap forward. It could take other companies many years to reach that same stage of progress.

The Need for Collective Action on Responsible AI Development

The above sections help establish that it is beneficial to develop safe and socially secure AI systems and that the benefits of such systems would outweigh the costs. However, in the current scenario where competition among companies to develop products is extremely fierce. This can lead to the companies to undermine the safety and invest lesser resources to ensure that the systems are developed responsibly.
To understand how much a company may be willing to corporate on responsible AI development, the authors propose a payoff matrix of a cooperate-defect game. An excerpt from the paper further defines the matrix:
"Consider the payoff matrix of a cooperate-defect game in which two agents (AI companies) can cooperate (develop responsibly) or defect (fail to develop responsibly). Here the first letter in each pair represent the expected payoff for Agent 1, and the second letter in each pair represents the payoff for Agent 2.
CooperateDefect
Cooperatea1,a2a_1,a_2b1,b2b_1,b_2
Defectc1,c2c_1,c_2d1,d2d_1,d_2

*Let pp be the probability that Agent 1 assigns to Agent 2 cooperating and let qq be the probability that Agent 2 assigns to Agent 1 cooperating. We assume it is rational for Agent 1 to cooperate if the expected value of cooperation (the likelihood Agent 2 will cooperate times $$a_1 plus the likelihood Agent 2 will defect times b1) is greater than the expected value of defection (the likelihood Agent 2 will cooperate times c1 plus the likelihood Agent 2 will defect times d1). We assume the same is true of Agent 2."
The authors of the paper see responsible AI development as a collective action problem. They identify five key factors that will encourage collaboration and cooperation.
  1. High Trust: Being more confident that others will cooperate (p,q)(p,q)
  2. Shared Upside: Assigning a higher expected value to mutual cooperation (a1,a2a_1,a_2)
  3. Low Exposure: Assigning a lower expected cost to unreciprocated cooperation (b1,c2b_1,c_2)
  4. Low Advantage: Assigning a lower expected value to not reciprocating cooperation (c1,b2c_1,b_2)
  5. Shared Downside: Assigning a lower expected value to mutual defection (d1,d2d_1,d_2)

Strategies to Improve AI Industry Cooperation on Safety

The authors identify four strategies that can increase cooperation on safety since they have high benefits based on the factors mentioned above.
  1. Promote accurate beliefs about the opportunities for cooperation
  2. It is always beneficial to correct misconceptions to improve and enhance cooperation. These misconceptions include that safety and security risks can be ignored.
  3. Collaborate on shared research and engineering challenges
  4. The authors explain that collaborating on research can provide high benefits that include technical insights, stabilizing expectations regarding who is working on what via public information about joint investments as well as interpersonal dialogue, concretizing the shared upside of AI, and establishing more beneficial publications and deployment decisions.
  5. Open up more aspects of AI development to appropriate oversight and feedback
  6. Incentivize adherence to high standards of safety
  7. Additional incentives can always be added to the incentives available today or to those that may exist in the future. These include:
    Social incentives like criticizing or appreciating specific decisions can help chance the practices and perceptions of other companies.Economic incentives by governments, industries, or organizations can encourage the adoption of specific norms.Legal incentives by penalizing companies following malpractice. Domain-specific incentives like providing early access to the latest computer hardware.

Conclusion

With the increased pace of AI development, it is critical to ensure that the technologies we create do not have a negative impact on society. The uncertain nature and breakneck pace at which AI is being developed calls for stronger regulations that ensure the safety and well-being of all consumers. Hence, we need to find and develop ways to build trust and encourage cooperation among individuals and organizations inventing the generation of AI applications.
What are your thoughts on responsible AI development and its impact on society? Do you have any questions for Miles? Write them down in the comments below!

Tags: Beginner, NLP, OpenAI, Case Study, GPT

Created with ❤️ on Weights & Biases.

https://wandb.ai/wandb/gallery/reports/An-Interview-With-Miles-Brundage-From-OpenAI--VmlldzoyNDU5MDk
Made with Weights & Biases. Sign up or log in to create reports like this one.
Iterate on AI agents and models faster. Try Weights & Biases today.
Sign upTry W&B now