Professor Nick Bostrom is Honorary/Advisory Philosopher of the United Sigma Intelligence Association. He is a Professor at Oxford University, where he leads the Future of Humanity Institute as its founding director.
Nick Bostrom is a Swedish-born philosopher and polymath with a background in theoretical physics, computational neuroscience, logic, and artificial intelligence, as well as philosophy. He is a Professor at Oxford University, where he leads the Future of Humanity Institute as its founding director. (The FHI is a multidisciplinary university research centre; it is also home to the Centre for the Governance of Artificial Intelligence and to teams working on AI safety, biosecurity, macrostrategy, and various other technology or foundational questions.) He is the author of some 200 publications, including Anthropic Bias (2002), Global Catastrophic Risks (2008), Human Enhancement (2009), and Superintelligence: Paths, Dangers, Strategies (2014), a New York Times bestseller which helped spark a global conversation about artificial intelligence. Bostrom’s widely influential work, which traverses philosophy, science, ethics, and technology, has illuminated the links between our present actions and long-term global outcomes, thereby casting a new light on the human condition.
He is recipient of a Eugene R. Gannon Award, and has been listed on Foreign Policy’s Top 100 Global Thinkers list twice. He was included on Prospect’s World Thinkers list, the youngest person in the top 15. His writings have been translated into 28 languages, and there have been more than 100 translations and reprints of his works. He is a repeat TED speaker and has done more than 2,000 interviews with television, radio, and print media. As a graduate student he dabbled in stand-up comedy on the London circuit, but he has since reconnected with the doom and gloom of his Swedish roots.
My interests cut across many disciplines and may therefore at the surface appear somewhat scattered, but they all reflect a desire to figure out how to orient ourselves with respect to important values. I refer to this as “macrostrategy”: the study of how long-term outcomes for humanity may be connected to present-day actions. My research seeks to contribute to this by answering particular sub-questions or by developing conceptual tools that help us think about such questions more clearly.
A key part of the challenge is often to notice that a problem even exists – to find it, formulate it, and then make enough initial progress in understanding it to let us break it into more tractable components and research tasks. Much of my work (and that of the Future of Humanity Institute) operates in such a pre-paradigm environment. We tend to work on problems that the rest of academia ignores either because the problems are not yet recognized as important or because it is unclear how one could conceivably go about doing research on them; and we try to advance understanding of them to the point where it becomes possible for a larger intellectual community to engage with them productively. For example, a few years ago, AI alignment fell into this category: hardly anybody thought it was important, and it seemed like the kind of thing a science fiction author might write novels about but that there was no way to study scientifically. By now, it has emerged as a bona fide research field, with people writing code and equations and making incremental progress. Significant cognitive work was required to get to this point.
I have also originated or contributed to the development of ideas such as the simulation argument, existential risk, transhumanism, information hazards, superintelligence strategy, astronomical waste, crucial considerations, observation selection effects in cosmology and other contexts of self-locating belief, anthropic shadow, the unilateralist’s curse, the parliamentary model of decision-making under normative uncertainty, the notion of a singleton, the vulnerable world hypothesis, along with a number of analyses of future technological capabilities and concomitant ethical issues, risks, and opportunities.
Technology is a theme in much of my work (and that of the FHI) because it is plausible that the long-term outcomes for our civilization depend sensitively on how we handle the introduction of certain transformative capabilities. Machine intelligence, in particular, is a big focus. We also work on biotechnology (both for its human enhancement applications and because of biosecurity concerns), nanotechnology, surveillance technology, and a bunch of other potential developments that could alter fundamental parameters of the human condition.
There is a “why” beyond mere curiosity behind my interest in these questions, namely the hope that insight here may produce good effects. In terms of directing our efforts as a civilization, it would seem useful to have some notion of which direction is “up” and which is “down”—what we should promote and what we should discourage. Yet regarding macrostrategy, the situation is far from obvious. We really have very little clue which of the actions available to present-day agents would increase or decrease the expected value of the long-term future, let alone which ones would do so the most effectively. In fact, I believe it is likely that we are overlooking one or more crucial considerations: ideas or arguments that might plausibly reveal the need for not just some minor course adjustment in our endeavours but a major change of direction or priority. If we have overlooked even just one such crucial consideration, then all our best efforts might be for naught—or they might even be making things worse. Those seeking to make the world better should therefore take it as important to get to the bottom of these matters, or else to find some way of dealing wisely with our cluelessness if it is inescapable.
The FHI works closely with the effective altruism community (e.g., we share office space with the Centre for Effective Altruism) as well as with AI leaders, philanthropic foundations, and other policymakers, scientists, and organizations to ensure that our research has impact. These communication efforts are sometimes complicated by information hazard concerns. Although many in the academic world take it as axiomatic that discovering and publishing truths is good, this assumption may be incorrect; certainly it may admit of exceptions. For instance, if the world is vulnerable in some way, it may or may not be desirable to describe the precise way it is so. I often feel like I’m frozen in an ice block of inhibition because of reflections of this sort. How much easier things would be if one could have had a guarantee that all one’s outputs would be either positive or neutral, and one could go full blast!
ETHICS & POLICY
What properties should we want a proposal for an AI governance pathway to have?[w/ Allan Dafoe & Carrick Flynn] [v. 4.3 (2018)] [forthcoming in Liao, S.M. (ed.): Ethics of Artificial Intelligence (Oxford University Press)] [pdf]
Recounts the Tale of a most vicious Dragon that ate thousands of people every day, and of the actions that the King, the People, and an assembly of Dragonologists took with respect thereto.[J Med Ethics, Vol. 31, No. 5 (2005): 273–277] [translations: Czech, Dutch, Finnish, French, German, Hebrew, Italian, Russian, Slovenian, Spanish, Chinese, Polish][Animation] [html] [pdf] [mp3]
We present a heuristic for correcting for one kind of bias (status quo bias), which we suggest affects many of our judgments about the consequences of modifying human nature. We apply this heuristic to the case of cognitive enhancements, and argue that the consequentialist case for this is much stronger than commonly recognized.[w/ Toby Ord] [Ethics, Vol. 116, No. 4 (2006): 656–680] [pdf]
An analysis of the global desirability of different forms of openness (including source code, science, data, safety techniques, capabilities, and goals).[Global Policy, Vol. 8, Iss. 2 (May 2017): 135–148] [html] [pdf]
Suns are illuminating and heating empty rooms, unused energy is being flushed down black holes, and our great common endowment of negentropy is being irreversibly degraded into entropy on a cosmic scale. These are resources that an advanced civilization could have used to create value-structures, such as sentient beings living worthwhile lives…[Utilitas, Vol. 15, No. 3 (2003): 308–314] [translations: Russian, Portuguese] [html] [pdf]
Cosmology shows that we might well be living in an infinite universe that contains infinitely many happy and sad people. Given some assumptions, aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. But you can presumably do only a finite amount of good or bad. Since an infinite cardinal quantity is unchanged by the addition or subtraction of a finite quantity, it looks as though you can’t change the value of the world. Aggregative consequentialism (and many other important ethical theories) are threatened by total paralysis. We explore a variety of potential cures, and discover that none works perfectly and all have serious side-effects. Is aggregative ethics doomed?[Analysis and Metaphysics, Vol. 10 (2011): 9–59] [Original draft was available in 2003] [html] [pdf]
In cases where several altruistic agents each have an opportunity to undertake some initiative, a phenomenon arises that is analogous to the winner’s curse in auction theory. To combat this problem, we propose a principle of conformity. It has applications in technology policy and many other areas.[Nick Bostrom, Thomas Douglas & Anders Sandberg (2016) The Unilateralist’s Curse and the Case for a Principle of Conformity, Social Epistemology, 30:4, 350–371, DOI: 10.1080/02691728.2015.1108373] [pdf]
Does human enhancement threaten our dignity as some have asserted? Or could our dignity perhaps be technologically enhanced? After disentangling several different concepts of dignity, this essay focuses on the idea of dignity as a quality (a kind of excellence admitting of degrees). The interactions between enhancement and dignity as a quality are complex and link into fundamental issues in ethics and value theory.[Human Dignity and Bioethics: Essays Commissioned by the President’s Council on Bioethics (Washington, D.C., 2008): 173–207] [pdf]
Brief paper, critiques a host of bioconservative pundits who believe that enhancing human capacities and extending human healthspan would undermine our dignity.[Bioethics, Vol. 19, No. 3 (2005): 202–214] [translations: Italian, Slovenian, Portuguese] [Was chosen for inclusion in a special anthology of the best papers published in this journal in the past two decades] [html] [pdf]
Original essays by various prominent moral philosophers on the ethics of human enhancement.[Eds. Nick Bostrom & Julian Savulescu (Oxford University Press, 2009)]
The introductory chapter from the book: 1–22[w/ Julian Savulescu] [pdf]
A transhumanist ethical framework for public policy regarding genetic enhancements, particularly human germ-line genetic engineering[Journal of Value Inquiry, Vol. 37, No. 4 (2003): 493–506] [html] [pdf]
Overview of ethical issues raised by the possibility of creating intelligent machines. Questions relate both to ensuring such machines do not harm humans and to the moral status of the machines themselves.[w/ Eliezer Yudkowsky] [Cambridge Handbook of Artificial Intelligence, eds. William Ramsey & Keith Frankish (Cambridge University Press)] [translation: Portuguese] [pdf]
Some cursory notes; not very in-depth.[Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Vol. 2, ed. I. Smit et al., Int. Institute of Advanced Studies in Systems Research and Cybernetics, 2003, 12–17] [translation: Italian] [html] [pdf]
Short article summarizing some of the key issues and offering specific recommendations, illustrating the opportunity and need for “smart policy”: the integration into public policy of a broad-spectrum of approaches aimed at protecting and enhancing cognitive capacities and epistemic performance of individuals and institutions.[w/ Rebecca Roache] [Enhancing Human Capacities, eds. J. Savulescu, R. ter Muelen, & G. Kahane (Wiley-Blackwell, 2011)] [pdf]
After some definitions and conceptual clarification, I argue for two theses. First, some posthuman modes of being would be extremely worthwhile. Second, it could be good for human beings to become posthuman.[Medical Enhancement and Posthumanity, eds. Bert Gordijn and Ruth Chadwick (Springer, 2008): 107–137] [pdf]
The revised version 2.1. The document represents an effort to develop a broadly based consensus articulation of the basics of responsible transhumanism. Some one hundred people collaborated with me in creating this text. Feels like from another era.[translations: German, Hungarian, Dutch, Russian, Polish, Finnish, Greek, Italian] [pdf]
Wonderful ways of being may be located in the “posthuman realm”, but we can’t reach them. If we enhance ourselves using technology, however, we can go out there and realize these values. This paper sketches a transhumanist axiology.[Ethical Issues for the 21st Century, ed. Frederick Adams, Philosophy Documentation Center Press, 2003; reprinted in Review of Contemporary Philosophy, Vol. 4, May (2005)] [translations: Polish, Portuguese, Spanish] [html] [pdf]
The human desire to acquire new capacities, to extend life and overcome obstacles to happiness is as ancient as the species itself. But transhumanism has emerged gradually as a distinctive outlook, with no one person being responsible for its present shape. Here’s one account of how it happened.[Journal of Evolution and Technology, Vol. 14, No. 1 (2005)] [translation: Spanish] [pdf]
RISK & THE FUTURE
Discusses the Fermi paradox, and explains why I hope we find no signs of life, whether extinct or still thriving, on Mars or anywhere else we look.[Technology Review, May/June issue (2008): 72–77] [translation: Italian] [pdf]
Existential risks are those that threaten the entire future of humanity. This paper elaborates the concept of existential risk and its relation to basic issues in axiology and develops an improved classification scheme for such risks. It also describes some of the theoretical and practical challenges posed by various existential risks and suggests a new way of thinking about the ideal of sustainability.[Global Policy, Vol. 4, Iss. 3 (2013): 15–31] [translation: Portuguese] [html] [pdf]
Examines the risk from physics experiments and natural events to the local fabric of spacetime. Argues that the Brookhaven report overlooks an observation selection effect. Shows how this limitation can be overcome by using data on planet formation rates.[w/ Max Tegmark] [expanded; Nature, Vol. 438 (2005): 754] [translation: Russian] [pdf]
This paper discusses four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity.[New Waves in Philosophy of Technology, eds. Jan-Kyrre Berg Olsen, Evan Selinger & Soren Riis (Palgrave McMillan, 2009) [pdf] [html]
Twenty-six leading experts look at the gravest risks facing humanity in the 21st century, including natural catastrophes, nuclear war, terrorism, global warming, biological weapons, totalitarianism, advanced nanotechnology, general artificial intelligence, and social collapse. The book also addresses overarching issues—policy responses and methods for predicting and managing catastrophes. Foreword by Lord Martin Rees.[Eds. Nick Bostrom & Milan Cirkovic (Oxford University Press, 2008)]. Introduction chapter free here [pdf]
This paper explores some dystopian scenarios where freewheeling evolutionary developments, while continuing to produce complex and intelligent forms of organization, lead to the gradual elimination of all forms of being worth caring about. We then discuss how such outcomes could be avoided and argue that under certain conditions the only possible remedy would be a globally coordinated effort to control human evolution by adopting social policies that modify the default fitness function of future life forms.[Death and Anti-Death, ed. Charles Tandy (Ria University Press, 2005): 339–371] [pdf] [html]
Technological revolutions are among the most important things that happen to humanity. This paper discusses some of the ethical and policy issues raised by anticipated technological revolutions, such as nanotechnology.[Nanoscale: Issues and Perspectives for the Nano Century, eds. Nigel M. de S. Cameron & M. Ellen Mitchell (John Wiley, 2007): 129–152] [pdf]
Existential risks are ways in which we could screw up badly and permanently. Remarkably, relatively little serious work has been done in this important area. The point, of course, is not to welter in doom and gloom but to better understand where the biggest dangers are so that we can develop strategies for reducing them.[Journal of Evolution and Technology, Vol. 9, No. 1 (2002)] [html] [pdf] [translations: Russian, Belarusian]
Information hazards are risks that arise from the dissemination or the potential dissemination of true information that may cause harm or enable some agent to cause harm. Such hazards are often subtler than direct physical threats, and, as a consequence, are easily overlooked. They can, however, be important.[Review of Contemporary Philosophy, Vol. 10 (2011): 44–79 (first version: 2009)] [pdf]
Concept describing a kind of social structure.[Linguistic and Philosophical Investigations, Vol. 5, No. 2 (2006): 48–54] [translation: Polish]
The embryo selection during IVF can be vastly potentiated when the technology for stem-cell derived gametes becomes available for use in humans. This would enable iterated embryo selection (IES), compressing the effective generation time in a selection program from decades to months.[w/ Carl Shulman] [Global Policy, Vol. 5, Iss. 1 (2014): 85–92] [pdf]
Some have argued that because blind evolutionary processes produced human intelligence on Earth, it should be feasible for clever human engineers to create human-level artificial intelligence in the not-too-distant future. We evaluate this argument.[w/ Carl Shulman] [J. Consciousness Studies, Vol. 19, No. 7–8 (2012): 103–130] [pdf]
Human beings are a marvel of evolved complexity. Such systems can be difficult to enhance. Here we describe a heuristic for identifying and evaluating the practicality, safety and efficacy of potential human enhancements, based on evolutionary considerations.[w/ Anders Sandberg] [Enhancing Humans, eds. Julian Savulescu & Nick Bostrom (Oxford University Press, 2009): 365–416 [pdf]
Presents two theses, the orthogonality thesis and the instrumental convergence thesis, that help understand the possible range of behavior of superintelligent agents—also pointing to some potential dangers in building such an agent.[Minds and Machines, Vol. 22, Iss. 2 (May 2012): 71–85] [pdf] [translation: Portuguese]
A 130-page report on the technological prerequisites for whole brain emulation (aka “mind uploading”).[w/ Anders Sandberg] [Technical Report #2008–3, Future of Humanity Institute, Oxford University (2008)] [pdf]
Cognitive enhancements in the context of converging technologies.[w/ Anders Sandberg] [Annals of the New York Academy of Sciences, Vol. 1093 (2006): 201–207] [pdf]
Some new ideas related to the challenge of endowing a hypothetical future superintelligent AI with values that would cause it to act in ways that are beneficial. Paper is somewhat obscure.[December, 2014] [pdf]
Game theory model of a technology race to develop AI. Participants skimp on safety precautions to get there first. Analyzes factors that determine level of risk in the Nash equilibrium.[w/ Stuart Armstrong & Carl Shulman] [Technical Report #2013–1, Future of Humanity Institute, Oxford University: 1–8] [AI and Society, Vol. 31, No.2 (2016): 201–206] [pdf]
Preliminary survey of various issues related to the idea of using boxing methods to safely contain a superintelligent oracle AI.[w/ Stuart Armstrong & Anders Sandberg] [Minds and Machines, Vol. 22, No. 4 (2012): 299–324] [pdf]
Some polling data.[Fundamental Issues of Artificial Intelligence, ed. V. Müller (Springer, 2016)] [w/ Vincent Müller] [pdf]
Cognitive enhancement comes in many diverse forms. In this paper, we survey the current state of the art in cognitive enhancement methods and consider their prospects for the near-term future. We then review some of ethical issues arising from these technologies. We conclude with a discussion of the challenges for public policy and regulation created by present and anticipated methods for cognitive enhancement.[w/ Anders Sandberg] [Science and Engineering Ethics, Vol. 15 (2009): 311–341] [pdf]
This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct