Homo Deus. Краткая история будущего

Some experts and thinkers, such as Nick Bostrom, warn that humankind is unlikely to suffer this degradation, because once artificial intelligence surpasses human intelligence, it might simply exterminate humankind. The AI would likely do so either for fear that humankind would turn against it and try to pull its plug, or in pursuit of some unfathomable goal of its own. For it would be extremely difficult for humans to control the motivation of a system smarter than themselves.
Even preprogramming the system with seemingly benign goals might backfire horribly. One popular scenario imagines a corporation designing the first artificial super-intelligence and giving it an innocent test such as calculating pi. Before anyone realises what is happening, the AI takes over the planet, eliminates the human race, launches a campaign of conquest to the ends of the galaxy, and transforms the entire known universe into a giant super-computer that for billions upon billions of years calculates pi ever more accurately. After all, this is the divine mission its Creator gave it.21

In the past there were many things only humans could do. But now robots and computers are catching up, and may soon outperform humans in most tasks. True, computers function very differently from humans, and it seems unlikely that computers will become humanlike any time soon. In particular, it doesn’t seem that computers are about to gain consciousness and start experiencing emotions and sensations. Over the past half century there has been an immense advance in computer intelligence, but there has been exactly zero advance in computer consciousness. As far as we know, computers in 2016 are no more conscious than their prototypes in the 1950s. However, we are on the brink of a momentous revolution. Humans are in danger of losing their economic value because intelligence is decoupling from consciousness.

If Sapiens rule the world because we alone can cooperate flexibly in large numbers, then this undermines our belief in the sacredness of human beings. We tend to think that we are special, and deserve all kinds of privileges. As proof, we point to the amazing achievements of our species: we built the pyramids and the Great Wall of China; we deciphered the structure of atoms and DNA molecules; we reached the South Pole and the moon. If these accomplishments resulted from some unique essence that each individual human has – an immortal soul, say – then it would make sense to sanctify human life. Yet since these triumphs actually result from mass cooperation, it is far less clear why they should make us revere individual humans.

If you watch the National Geographic channel, go to a Disney film or read a book of fairy tales, you might easily get the impression that planet Earth is populated mainly by lions, wolves and tigers who are an equal match for us humans. Simba the lion king holds sway over the forest animals; Little Red Riding Hood tries to evade the Big Bad Wolf; and little Mowgli bravely confronts Shere Khan the tiger. But in reality, they are no longer there. Our televisions, books, fantasies and nightmares are still full of them, but the Simbas, Shere Khans and Big Bad Wolves of our planet are disappearing. The world is populated mainly by humans and their domesticated animals.