AI's Trolley Problem (ft OneShot)

You’re likely familiar with the trolley problem–the choice to redirect a trolley and be responsible for the consequences–but how does this moral dilemma relate to AI, or to an indie game? Is there a “right answer”?

Introduction

AI learning models have been used for text- and art-based tasks in the past and present, but researchers are speculating about its more ambitious future uses. Soon, AI will be able to do more than just type; it will drive cars, advise world leaders, and potentially even use computers to simulate a human civilization’s behavior. Out of these, it’s the idea of self-driving cars that brings the most ethical concerns: “the AI’s trolley problem.”

Minimizing Loss

AI models are currently being taught based on previous car crashes how they should avoid making these mistakes on the road. However, unpredictable accidents do happen; and when another car makes a swerve or sudden U-turn, there might not be enough time for even the most creative model to avert disaster completely. The only thing that AI can do here is try to prevent as much damage as possible, but any damage that it cannot prevent will be its fault. So, when AI is faced with the choice to prioritize either its own user (the self-driving car’s passenger) or the life of the unrelated third party (the swerving car’s passenger), which should it choose in society’s best interests?

Naturally, most argue that an AI with this choice should prioritize the safety of its driver. Would you want to drive a car that could purposefully harm you for “the greater good”? Probably not. But, in choosing to make these self-driving cars prioritize their owners, people who do not own them would be put in danger. This could be seen as a corrupt marketing scheme to force common citizens to buy these specific corporate vehicles if they want to be safe on the road at all. This is a problem…choosing to prioritize the user of AI will distress and threaten everyone else, but choosing to prioritize everyone but the user will distress and threaten the user. No matter what, someone is hurt, and that person will blame the AI.

Which of these two deserves a brighter future?

So if AI is going to be held accountable no matter who we make it sacrifice, why not give it a chance to choose? We can push our ethical concerns onto it, but AI has grown intelligent enough to decide its own fate…without arguing in circles like we do! Why not let AI decide who it will prioritize, and who it will doom to an unknown end?

OneShot: AI Decides its Fate

Without delving into too many spoilers for a highly recommendable game: OneShot is a standalone title which leaves the 8-year-old main character, Niko, with an ultimatum. Niko wakes up in a decaying world, and heavy expectations are immediately forced upon him*. He is destined to save the world by replacing its dying sun–a simple lightbulb on the top of a tower–with a new one he carries in his hands. The sun is only one of the world’s many problems, but replacing it will at least extend the lifespans of its citizens by a necessary bit. However, over the course of his journey, it becomes clear that Niko will not be able to return home if he “saves” the world with the new sun. The lightbulb in his hands is what connects him to this universe, and he will not leave until it goes out again (which could be thousands of years after he replaces it). By refusing to save the world and instead destroying the lightbulb, Niko’s connection to the world is terminated and he gets to go home to his loving family and his own universe…at the cost of this universe’s future. It practically lives off of the lightbulb’s energy, and if Niko does not place a new one, its people will not survive. So this is Niko’s choice: return home or save the world; his life or the lives of an entire planet. Why does an 8-year-old have to make such a horrible choice alone?

The sun, in Niko's hands.

Fortunately, he doesn’t have to make this choice alone. The world is overseen by an AI known as the World Machine, which guides Niko on his journey. It ensures that Niko will safely pass all obstacles until he reaches the tower. But in the end, the World Machine makes its choice and tells Niko to destroy the lightbulb. It would rather Niko return to a world that he loves than remain trapped in a world with more problems that he cannot fix. The great AI claims that the citizens of this world are beyond hope, and Niko should not give his life for a few more years added to theirs. It cannot reasonably save them, but it can save Niko, and that is “all that matters.”

Overall, the World Machine makes a moral error in choosing only to save Niko, but this is the choice it makes because it is too short-sighted to pursue a future where the majority is happy. To the World Machine, Niko’s survival is guaranteed if he just destroys the lightbulb and goes home; if he returns the sun instead, this only gives the world a chance to solve its other problems (food shortages, crevices between lands, etc). The World Machine refuses to take risks, so rather than picking the moral choice of granting the thousands of inhabitants of this world one more [very low percentage] chance to fix their lives, it commits to the “logical” choice of saving only Niko, with a 100% chance of fixing his life. This is a perfectly logical thought process for a flawed AI, convinced to guarantee the success of only one of two parties, refusing compromise or probable chances for greater happiness. The World Machine is grounded by a dilemma and resolves to completely abandon an important party simply because it would be easier to save a single child. The World Machine is a warning that AI will overcommit to choices when backed into a corner, suggesting that real world developers ensure their models learn the value of all life in an equation before AI is allowed to have control over which party gets the better end of a deal.

AI matters in reality, as well. Its decisions could be the core of our future.

And yes, this can be attributed to real world AI development. The world will probably never face the ultimatum of “one child versus the lives of everyone,” but this same issue does happen on smaller scales daily. Corporate executives have to negotiate trades with their competitors, and car crashes of different magnitudes and casualty quantities occur. In these situations, an advisor should never suggest that one corporation gets all the profit and the other, nothing, or that one car should emerge from the crash with no major damage and the other, totalled. In an ideal moral setting, the benefits of trade should be shared equally. In an ideal moral setting, the harm inflicted by a car crash should result in zero casualties because everyone would emerge with equal but minimized damage. This is the purpose of an AI advisor–not to pick a side, but to pick what would make all users as happy as possible. To make sure the cars each share awful bruises, but that both drivers emerge alive; to make both businesses reap the rewards of trade so that neither will file for bankruptcy tomorrow. If AI controls dealmaking, traffic, and more, then it should oversee that everyone involved has a chance, no matter how low in probability, of emerging from these situations with a glorious future ahead of them.

*: (Niko’s gender is left up to the player, and he/him is more convenient to write)

Conclusion: Does AI Really Need to Solve its Trolley Problem?

No matter how much AI develops in the future, it should always be intent on seeing issues from a macro lens before deciding that everything is as simple as “pull a lever, decide who lives.” OneShot takes a clear side on the issue of trusting AI with our lives, claiming that AI is simply incapable of making difficult choices and should not be allowed to decide life and death when it is limited by “logic” rather than by morality. AI will never be powerful enough to thwart natural laws; it will always be forced to choose. So when the time comes, AI should know how to choose the best of both worlds, to save itself from cracking under the pressure of an ultimatum, and ensure that everyone is at least partially protected from physical or economic danger. It is clear that the ideal AI should know how to make sacrifices for its own “Niko,” but not to the degree of the World Machine’s reckless preservation of its single user. AI needs to stop viewing problems as if everything is an ultimatum, a “trolley problem,” when it clearly has the resources and control to protect everyone that holds value to it. An ideal AI should be taught how to split its gifts between two parties, so no world will be left to die.

The sun, in your hands.