Inside Effective Altruism’s plan to stop science killing us all

Effective Altruism (EA), a movement of rationalist do-gooders that has been growing in size and influence for just over a decade, hit the headlines worldwide in 2022 – although not quite as its supporters hoped.

EA advocates use cold reason to decide how and who gets philanthropic aid, and how technologies affect that calculus in problems ranging from preventing hunger to preventing human annihilation. Yet it really entered public consciousness for rather less noble reasons last November when one of its biggest advocates and donors, Sam Bankman-Fried, fell from grace after his cryptocurrency exchange FTX collapsed, leaving depositors empty handed. He now faces fraud charges and a possible lengthy jail sentence.

But the EA movement continues, and fits into a growing concern about the sometimes bad consequences of emerging technologies. As the Bankman-Fried episode suggests, EA has attracted a lot of money from the ultra-rich; and some of its advocates are now using that philanthropic war chest to steer the direction of research and scientific policy through grants and think tanks. In brief, EA theorists – clustered around Oxford University – want to rein in and reshape research and innovation so that new breakthroughs don’t inadvertently wipe out the human race.

This dovetails with a broader effort, among policy makers in Washington, Paris, Berlin and Brussels, to do a better job of analysing the potential impact of innovations, or “technology assessment”, an older idea now making a comeback as the catastrophic potential of technology – through climate change, lab leaks and nuclear weapons – is once again in the headlines.

“In this 21st century that we that we live in, technologies are becoming ever more powerful,” Jonas Sandbrink, a biosecurity researcher at Oxford University, told Science|Business.“Given we are facing these extremely powerful technologies, the question is, what can we do to make sure that they go well?”

Sandbrink is co-author of a new paper advocating for “differential technology development”, a concept the EA community hopes will prevent or mitigate risky new technologies, particularly catastrophic ones, from emerging from the lab.

The idea is to use policy, regulatory and funding tools to make sure that technology to make us safe emerges before, or alongside, technology that can pose a risk. EA’s careers advice service argues that “differential development” is on the whole preferable to “indiscriminate progress” in science and technology.

Sandbrink’s co-authors include the head of long-term strategy and governance at DeepMind, one of the world’s leading artificial intelligence companies, and a policy adviser to the Organisation for Economic Cooperation and Development.

Critics are worried that EA’s focus on the apocalyptic potential of science could stymie the positive potential of R&D progress. They also fear it risks ignoring less catastrophic, but still harmful effects of new technologies that are causing damage right now.

This matters because, despite the FTX scandal, there is support for EA notions like differential technology development in the research and policy community.

“I think the time is ripe for a structural rethink of the way in which we design technology, at least in academia,” said Andrea Renda, a senior research fellow and specialist in digital governance at the Centre for European Policy Studies, a think tank based in Brussels.

Last November, Renda helped organise the inaugural Global PeaceTech Conference in Florence, which discussed similar ideas to those proposed by Sandbrink and his co-authors.

What we owe the future

In order to understand how EA might reshape science, it’s worth grasping the sheer extent of its current influence. Before the FTX scandal, 2022 was a breakthrough year for the movement. The Oxford ethicist Will MacAskill, one of the intellectual leaders of EA, was profiled in the New Yorker  and appeared on the Daily Show to plug his book “What We Owe the Future”, a call to safeguard humanity’s long-term prospects.

The movement emerged from a knot of moral philosophers based at Oxford, and draws on the ideas of Princeton bioethicist Peter Singer, who makes the case for giving away large parts of your income to help the needy, regardless of where in the world they live.

EA now boasts more than 80 chapters meeting in cities across the world and has managed to convince more than 1,000 people to change their careers through its advice service. EA ideas have started gaining influence in other areas, like foreign policy.

And it has money to spend – a lot of money. In 2021, Benjamin Todd, a prominent figure in EA, calculated the movement had $46 billion at its disposal. About a third of that was from Bankman-Fried, but though he is now on $250 million bail, the movement is still flush with cash. Its biggest donors by far are Facebook co-founder Dustin Moskovitz and his partner Cari Tuna, while Tesla and Twitter CEO Elon Musk has also offered the EA ecosystem money and public support.

The movement is already impacting research through private philanthropy, ploughing hundreds of millions of research dollars a year into areas like reducing existential risks, global health, and animal suffering. Moskovitz and Tuna are the chief funders behind Open Philanthropy, an EA grant-making vehicle than earmarked $400 million of awards in 2021. It has given tens of millions of dollars to new US-based science and technology think tanks, including the Center for Security and Emerging Technology and the Institute for Progress.

EA also takes a rather niche interest in what it thinks are neglected technologies that might save a lot of people in a disaster. The movement’s careers advice podcast recently devoted a three-hour episode on how to survive a nuclear winter by farming seaweed or turning wood into sustaining gruel. 

Of course, its rise to prominence has attracted plenty of criticism. Detractors see the movement as cosying up to billionaires, elitist, too uncritical of modern capitalism, and overly focused on the hazy far-term future of humanity rather than on human suffering here and now.

These criticisms were turbo-charged by the downfall of Bankman-Fried, known as SBF, who is accused of using customer deposits at FTX to bail out his separate investment fund. (He has pleaded not guilty to US criminal charges.) SBF had been seen as the most successful example of EA’s controversial “earning to give” philosophy, whereby EA followers get rich in order to give back to causes the movement deems worthy. It’s unclear if Bankman-Fried was a true believer in the movement, amassing profits from crypto for what he believed was the greater good, or if his largesse was an elaborate case of reputation washing.

But although the EA movement now faces a painful internal reckoning over why it so readily embraced a crypto billionaire with money to burn, its followers show no sign of packing up and going home.

Making altruism effective

To understand why the EA movement wants to reshape science and technology, it’s necessary to delve into its history, and how its underlying philosophy has evolved. There is in this history, as one would expect, a lot of fancy logic and academic speculation; but it’s stirring debate today in the real world of technology assessment and regulation.

In the first half of the 2010s, the movement was heavily focused on injecting empirical rigour into the charity sector – in other words, making altruism as “effective” as possible.

Organisations such as US-based GiveWell continue this work to the present day, researching which charities save the most lives per dollar (currently, malaria prevention, vitamin A deficiency and vaccination are top priorities). Even today, much EA community discussion is about which charities are most worth backing. Plenty of EA money still flows to this fairly standard charity work in global health and development.

But in recent years, much of the EA community has become enamoured with a much grander-sounding - and more controversial - philosophy.

Is science lethal?

“Longtermism” rests on the belief that right now, human civilisation may only be in its infancy. Given our stunning technological and social advances over a mere few centuries, if humanity can continue to survive and prosper, most of our history is ahead of us.

Humanity may even expand to colonise the galaxy, filling the stars with trillions of humans living lives of currently unimaginable abundance, fulfilment and joy, so the argument goes. In other words, because most moral value lies in the longterm future, it is of utmost important to safeguard it however we can.

An obvious objection is that it’s incredibly hard to know what actions now will produce a flourishing civilisation in a thousand years’ time, let alone a million.  But there is one rather obvious thing we can do this century to protect our future: not wipe ourselves out.

Of course, almost no one wants humanity to go extinct. But for longtermists and much of the EA movement, extinction, or some kind of irreversible civilizational collapse, would be a particularly unfathomable tragedy, because it would snuff out potentially trillions of happy lives in the future, and possibly the only conscious life in the universe.

In 2020, another Oxford philosopher and EA leader, Toby Ord, published “The Precipice”, in which he argued mankind had about a one in six chance of going extinct over the next century. Ord thinks that natural risks – like an asteroid hitting earth – have almost no chance of wiping out mankind. Even climate change and nuclear war, despite causing catastrophic destruction, are also extremely unlikely to completely kill us off.

This is naturally a highly contested line of argument: such a longtermist approach, in the eyes of critics, reduces cataclysms like climate change or the Second World War to mere blips in the vast potential story of humanity, downplaying the causes of terrible human suffering just because they don’t present a truly existential thread.

Destroying the world from a lab

Significantly for scientists, Ord instead believes that the biggest existential risks to mankind will emerge from labs.

Top of the threat list is “unaligned artificial intelligence”, whereby researchers create AI so smart that it escapes our control, and accidentally wipes out or subjugates humanity in the pursuit of some poorly programmed goal. This fear has begun to creep into official policy: the UK government said it took the risk of unaligned artificial general intelligence “seriously” in a new AI strategy last year. 

Engineered pandemics, whereby advances in bioscience allow the creation of unimaginably deadly and contagious pathogens, also have about a one in 30 risk of killing off mankind this century, Ord thinks, and as yet unknown scientific breakthroughs pose a similar level of risk (unsurprisingly, many quibble with these precise estimates).

You don’t have to be a longtermist to worry about runaway AI and engineered pandemics, and Sandbrink’s paper doesn’t explicitly mention the philosophy. Both these technologies are expected to become growing problems over the coming decades, not centuries, and could do plenty of damage short of killing off mankind. Plus, not all EA advocates are longtermists, but worry that the two have become synonymous.

But longtermism has undoubtedly put catastrophic new technology at the centre of EA concerns. EA’s career advice service advises graduates that AI risk and lab-created pandemics are the top two most important fields to enter if they want to make the world a better place.  

Differential technology development

The EA solution to the existential risk-generating nature of science is “differential technology development”, a concept that regularly bubbles up in the movement’s discussions.

It was first coined by yet another Oxford philosopher, Nick Bostrom, one of the first to raise alarm bells about runaway AI in his 2014 book, “Superintelligence”. Bostrom is known, among other things, for advancing a haunting solution to the Fermi Paradox, the conundrum of why we appear alone in the universe despite the existence of billions of earth-like planets. Alien civilisations, he speculates, always discover a doomsday technology that wipes them out before they can percolate throughout the galaxy (Bostrom has also recently been under fire for racist comments he made as a student, for which he has apologised).

In their new paper, Sandbrink and his co-authors have tried to put flesh on the bones of differential technology development, and spell out what it might mean in practice.

For example, the invention by the US in the 1960s of locks on nuclear weapons to prevent unauthorised firing have lowered the risk of accidental nuclear war, Sandbrink and his co-authors point out. But it would have been even better if they had been invented in the 1940s, because there were several decades when, Dr. Strangelove-style, a single rogue commander could have sparked a nuclear Armageddon.

Likewise, governments have backed R&D into solar panels, which are finally replacing dirty technologies like coal power stations and petrol-driven cars. But this begs the question of whether earlier solar R&D would have allowed a green energy revolution decades ago.

Sandbrink’s new paper urges policymakers and scientists to consider a whole suite of tools – including defunding certain research, voluntary moratoria, horizon scanning, plus prizes and other incentives to create safety-enhancing breakthroughs – to make sure good technology is in place before bad, or replaces bad technologies before they do too much damage. For example, bioscience funders could use a “dual use tiebreaker” to decide which projects to fund, prioritising those that are less likely to be turned towards military or offensive uses, Sandbrink argued.

Research funders need to take more of a “portfolio” approach, the paper stresses. This means if they are funding research that could lead to a risky technology, they should also be scanning for and investing in breakthroughs that could offset this danger too.

New old ideas

In a sense, these ideas are far from new. The attempt to steer technology so that it does more good than bad is as old as invention itself.

But Sandbrink thinks these ideas have fallen out of fashion in the research community, and he wants them back at the centre of policy. “We need to recommit to assessing technologies and thinking more strategically about how to advance them,” he argued.

During the Cold War, when the nuclear Sword of Damocles hung over the world, it was obvious that science had catastrophic potential, Sandbrink said. Scientists, horrified at the destruction of Hiroshima and Nagasaki by the fruits of modern physics, embarked on a period of introspection, creating, among other things, the Bulletin of the Atomic Scientists, which to this day warns the public and scientists about the risks of new technology. 

But from the 1990s onwards, this sense that science can be a kind of Pandora’s Box has arguably fallen by the wayside. In 1995, the widely respected Office of Technology Assessment, which since the mid-1970s provided the US Congress with detailed analysis of the promise and perils of new tech, was shut down in a Republican cost-cutting drive (Andrew Yang, a 2020 Democratic presidential candidate who has flirted with the Effective Altruism movement, has demanded that it be brought back).

Now, “there is this broad faith that knowledge is generally good,” said Sandbrink. “But I think we can probably expect that there's some knowledge that is net bad.”

Yet with the return of nuclear anxieties due to the Russian invasion of Ukraine, and the possibility that COVID-19 was the result of a lab leak in China, science’s catastrophic potential is once again top of people’s minds.

Wider support

David Winickoff, who heads the Organisation for Economic Cooperation and Development’s working party on bio, nano and converging technology, told Science|Business he is supportive of many of the ideas Sandbrink and his co-authors are pushing. 

“While technology may have an inner developmental logic, in fact human agency and therefore policy can have a role in shaping its trajectory,” he said. The OECD is trying to help governments manage new technologies so that they tip in society’s favour, he added.

What worries Winickoff, however, is that too much focus on the catastrophic risks of new technology might overshadow more mundane, but still harmful side effects, like AI disrupting the labour force. It might also “stoke undue fears about emerging technologies”, chilling useful breakthroughs. 

Technology assessment

Meanwhile in the US, the spirit of the Office of Technology Assessment has been revived to deal with an onrush of disruptive new technologies. This is part of a shift in recent years by governments to take the risks of new technology more seriously, paralleling the EA movement’s warnings about lab-induced disaster.

Since 2019, the newly established Science, Technology Assessment, and Analytics team, based in the Government Accountability Office, has issued around 50 assessments and explainers for members of Congress, and now has more than 120 staff running the rule over new technologies. “The pendulum swung back” towards the need to again do technology assessment after a hiatus following the end of the Cold War, said Timothy Persons, the organisation’s chief scientist.

What’s made this so pressing is the “convergence” of new technologies, he explains, with new breakthroughs, powerful even in isolation, combining to upend the labour market and economy. His team has run analyses on technologies including messenger RNA vaccines (the key COVID-19 vaccine technology), antimicrobial resistance, and 5G wireless networks.

But Persons stresses that his focus is not on catastrophic, extinction-level risks to the extent that some in the EA movement might wish. “There are apocalyptic narratives on every technology I can imagine,” he said. “But risk is really a probability times consequence […] and sometimes [for] that catastrophic narrative, the probability is quite low.”

And like the OECD’s Winickoff, he worries that too much of a catastrophic risk narrative could stymie the potential benefits of technologies like AI.

Risk is back

This tilt back towards technology assessment is happening in Europe too. Last September, lawmakers from Germany and 16 other countries, plus technology assessment officers from across Europe, gathered in Berlin to discuss how to cope with the overlapping crises – from energy shortages to climate change – that are disrupting society. Technology assessment is more important than ever, delegates were warned by Kai Gehring, chairman of the German Bundestag’s committee on education, research and technology development.

Armin Grunwald, head of the parliament’s own technology assessment organisation, said that after a post-Cold War period when new technology was largely mined for its potential benefits, the subject of risk was now firmly back on the agenda.

“After the end of Cold War, all these protection measures were no longer funded and they were ignored. Everybody thought it was the end of history, so there was no longer need seen to protect society against huge risks,” he said.

Now, Germany’s research ministry has started to shift some of its funding towards a “vulnerability and resilience based approach,” he said. Grunwald’s organisation, the Office of Technology Assessment at the Bundestag, has also begun cooperating with engineers at the Technical University of Delft in the Netherlands to engage in so-called “value sensitive design”, which seeks to embed ethics in technology.

But like his US counterpart, Grunwald is somewhat sceptical of EA’s particular emphasis on preventing existential risks. “We are not able in our projects to save humanity,” he said. A special focus on catastrophic risk is “valid”, he said, “but it does not orientate our daily work.”

Part of the difficulty, said Grunwald, is an obvious one: the impact of technology is down to how it is used by society, not just the properties of the technology itself. “The technological issues, they are part of the game, but only part - the other part is up to us humans,” he said. In other words, steering technology is a task for all of society, not just technologists.

A white guy problem?

This gets to the heart of criticisms of EA’s approach to science and technology. It’s seen by some as overly technocratic, and too concerned with the very worst risks, thereby skirting over more ordinary harms happening right now. 

“There's a real danger that by focusing on risk, and in particular catastrophic or existential risk, that you define the problem in very narrow terms,” said Jack Stilgoe, a science and technology studies professor at University College London, of Sandbrink’s paper. “And you ignore what people's actual concerns about technologies are.”

Focusing on an AI takeover risks neglecting the “more boring but far more profound” way in which the technology is already harming some people while benefiting others, he argued. For example, “where does a technology like facial recognition fit into this frame?”

Kate Crawford, a researcher into the social implications of AI at Microsoft, has catagorised Silicon Valley’s concern over runaway AI as a “white guy problem”.

Women and ethnic minorities are already being disadvantaged by discriminatory algorithms, she pointed out. But affluent white men choose to focus on AI’s existential risk because for them, that’s the only threat the technology actually poses, she wrote in 2016.

Still, Stilgoe thinks that it’s right that technology assessment is back on the agenda. Right now, government research funding is a kind of “fire and forget” model, he said. There are few expectations that scientists or funders anticipate the consequences of their research.

The idea of scientific foresight “has gone out of fashion a little bit,” Stilgoe said, “as government has slightly washed its hands of technology assessment. Which I think is a real shame. I think it's a really important role that government can and should play.”