Murder in virtual reality should be illegal

 You start by picking up the knife, or reaching for the neck of a broken-off bottle. Then comes the lunge and wrestle, the physical strain as your victim fights back, the desire to overpower him. You feel the density of his body against yours, the warmth of his blood. Now the victim is looking up at you, making eye contact in his final moments.

Science-fiction writers have fantasised about virtual reality (VR) for decades. Now it is here – and with it, perhaps, the possibility of the complete physical experience of killing someone, without harming a soul. As well as Facebook’s ongoing efforts with Oculus Rift, Google recently bought the eye-tracking start-up Eyefluence, to boost its progress towards creating more immersive virtual worlds. The director Alejandro G Iñárritu and the cinematographer Emmanuel Lubezki, both famous for Birdman (2014) and The Revenant (2015), have announced that their next project will be a short VR film.

But this new form of entertainment is dangerous. The impact of immersive virtual violence must be questioned, studied and controlled. Before it becomes possible to realistically simulate the experience of killing someone, murder in VR should be made illegal.

This is not the argument of a killjoy. As someone who has worked in film and television for almost 20 years, I am acutely aware that the craft of filmmaking is all about maximising the impact on the audience. Directors ask actors to change the intonation of a single word, while editors sweat over a film cut down to fractions of a second, all in pursuit of the right mood and atmosphere.

So I understand the appeal of VR, and its potential to make a story all the more real for the viewer. But we must examine that temptation in light of the fact that both cinema and gaming thrive on stories of conflict and resolution. Murder and violence are a mainstay of our drama, while single-person shooters are one of the most popular segments of the games industry.

The effects of all this gore are not clear-cut. Crime rates in the United States have fallen even as Hollywood films have become bloodier and violent video games have grown in popularity. Some research suggests that shooter games can be soothing, while other studies indicate they might be a causal risk factor in violent behaviour. (Perhaps, as for Frank Underwood in the Netflix series House of Cards (2013-), it’s possible for video games to be both those things.) Students who played violent games for just 20 minutes a day, three days in a row, were more aggressive and less empathetic than those who didn’t, according to research by the psychologist Brad Bushman at Ohio State University and his team. The repeated actions, interactivity, assuming the position of the aggressor, and the lack of negative consequences for violence, are all aspects of the gaming experience that amplify aggressive behaviour, according to research by the psychologists Craig Anderson at Iowa State University and Wayne Warburton at Macquarie University in Sydney. Mass shooters including Aaron Alexis, Adam Lanza and Anders Breivik were all obsessive gamers.

The problem of what entertainment does to us isn’t new. The morality of art has been a matter of debate since Plato. The philosopher Jean-Jacques Rousseau was skeptical of the divisive and corrupting potential of theatre, for example, with its passive audience in their solitary seats. Instead, he promoted participatory festivals that would cement community solidarity, with lively rituals to unify the jubilant crowd. But now, for the first time, technology promises to explode the boundary between the world we create through artifice and performance, and the real world as we perceive it, flickering on the wall of Plato’s cave. And the consequences of such immersive participation are complex, uncertain and fraught with risk.

Humans are embodied beings, which means that the way we think, feel, perceive and behave is bound up with the fact that we exist as part of and within our bodies. By hijacking our capacity for proprioception – that is, our ability to discern states of the body and perceive it as our own – VR can increase our identification with the character we’re playing. The ‘rubber hand illusion’ showed that, in the right conditions, it’s possible to feel like an inert prosthetic appendage is a real hand; more recently, a 2012 study found that people perceived a distorted virtual arm, stretched up to three times its ordinary length, to still be a part of their body.

It’s a small step from here to truly inhabiting the body of another person in VR. But the consequences of such complete identification are unknown, as the German philosopher Thomas Metzinger has warned. There is the risk that virtual embodiment could bring on psychosis in those who are vulnerable to it, or create a sense of alienation from their real bodies when they return to them after a long absence. People in virtual environments tend to conform to the expectations of their avatar, Metzinger says. A study by Stanford researchers in 2007 dubbed this ‘the Proteus effect’: they found that people who had more attractive virtual characters were more willing to be intimate with other people, while those assigned taller avatars were more confident and aggressive in negotiations. There’s a risk that this behaviour, developed in the virtual realm, could bleed over into the real one.

In an immersive virtual environment, what will it be like to kill? Surely a terrifying, electrifying, even thrilling experience. But by embodying killers, we risk making violence more tantalising, training ourselves in cruelty and normalising aggression. The possibility of building fantasy worlds excites me as a filmmaker – but, as a human being, I think we must be wary. We must study the psychological impacts, consider the moral and legal implications, even establish a code of conduct. Virtual reality promises to expand the range of forms we can inhabit and what we can do with those bodies. But what we physically feel shapes our minds. Until we understand the consequences of how violence in virtual reality might change us, virtual murder should be illegal.

Coding is not ‘fun’, it’s technically and ethically complex

 Programming computers is a piece of cake. Or so the world’s digital-skills gurus would have us believe. From the non-profit Code.org’s promise that ‘Anybody can learn!’ to Apple chief executive Tim Cook’s comment that writing code is ‘fun and interactive’, the art and science of making software is now as accessible as the alphabet.

Unfortunately, this rosy portrait bears no relation to reality. For starters, the profile of a programmer’s mind is pretty uncommon. As well as being highly analytical and creative, software developers need almost superhuman focus to manage the complexity of their tasks. Manic attention to detail is a must; slovenliness is verboten. Attaining this level of concentration requires a state of mind called being ‘in the flow’, a quasi-symbiotic relationship between human and machine that improves performance and motivation.

Coding isn’t the only job that demands intense focus. But you’d never hear someone say that brain surgery is ‘fun’, or that structural engineering is ‘easy’. When it comes to programming, why do policymakers and technologists pretend otherwise? For one, it helps lure people to the field at a time when software (in the words of the venture capitalist Marc Andreessen) is ‘eating the world’ – and so, by expanding the labour pool, keeps industry ticking over and wages under control. Another reason is that the very word ‘coding’ sounds routine and repetitive, as though there’s some sort of key that developers apply by rote to crack any given problem. It doesn’t help that Hollywood has cast the ‘coder’ as a socially challenged, type-first-think-later hacker, inevitably white and male, with the power to thwart the Nazis or penetrate the CIA.

Insisting on the glamour and fun of coding is the wrong way to acquaint kids with computer science. It insults their intelligence and plants the pernicious notion in their heads that you don’t need discipline in order to progress. As anyone with even minimal exposure to making software knows, behind a minute of typing lies an hour of study.

It’s better to admit that coding is complicated, technically and ethically. Computers, at the moment, can only execute orders, to varying degrees of sophistication. So it’s up to the developer to be clear: the machine does what you say, not what you mean. More and more ‘decisions’ are being entrusted to software, including life-or-death ones: think self-driving cars; think semi-autonomous weapons; think Facebook and Google making inferences about your marital, psychological or physical status, before selling it to the highest bidder. Yet it’s rarely in the interests of companies and governments to encourage us to probe what’s going on beneath these processes.

All of these scenarios are built on exquisitely technical foundations. But we can’t respond to them by answering exclusively technical questions. Programming is not a detail that can be left to ‘technicians’ under the false pretence that their choices will be ‘scientifically neutral’. Societies are too complex: the algorithmic is political. Automation has already dealt a blow to the job security of low-skilled workers in factories and warehouses around the world. White-collar workers are next in line. The digital giants of today run on a fraction of the employees of the industrial giants of yesterday, so the irony of encouraging more people to work as programmers is that they are slowly mobilising themselves out of jobs.

In an ever-more intricate and connected world, where software plays a larger and larger role in everyday life, it’s irresponsible to speak of coding as a lightweight activity. Software is not simply lines of code, nor is it blandly technical. In just a few years, understanding programming will be an indispensable part of active citizenship. The idea that coding offers an unproblematic path to social progress and personal enhancement works to the advantage of the growing techno-plutocracy that’s insulating itself behind its own technology.

Getting things moving

 The most important moments in invention are sometimes the imaginative leaps – even when they turn out to be dead-ends

Behind the drawn curtains of his home in Palo Alto, California, the railroad magnate Leland Stanford waited for his horse to be brought to life. A white sheet hung against one wall, and in the gloaming the only light came from a wood-and-brass construction at the back of the room. Suddenly, with a mechanical clatter and the hiss of an oxyacetylene lamp, a moving image appeared on the screen. It was little more than a silhouette, but Stanford and his astonished guests could clearly see Hawthorn, Stanford’s stallion, walking along as if it were right there in the room among them.

Eadweard Muybridge, proud and nervous, stood next to his device. The British photographer had a reputation in California thanks to his superb technical eye and his majestic Yosemite waterscapes, as well as the sensational murder of his wife’s lover five years before. But he had escaped conviction, and resumed work on a commission from Stanford to capture the motion and beauty of his benefactor’s beloved race horses.

As the applause from the audience died away, Stanford addressed the photographer. ‘I think you must be mistaken in the name of the animal,’ he said. ‘That is certainly not the gait of Hawthorn but of Anderson.’ It turned out that the stable staff at Stanford’s ranch had switched the horses. But so crisp was the outline, and so defined its movements, that Stanford could tell the difference.

This private demonstration for Stanford took place in the autumn of 1879, shortly after he bought the estate that would go on to become Stanford University. A hundred years later, Stanford and its surrounds would become renowned as the crucible of the ‘Silicon Valley’ computing boom, building on the ‘analytical engine’ first envisaged by another British inventor, the mathematician and polymath Charles Babbage.

Babbage and Muybridge were separated by class, generation and temperament. But for both creators, the path from conception to application for their technologies evolved in ways that they couldn’t have anticipated. Putting their lives side by side contains valuable insights about the contingency of history, and what it takes to be remembered as the ‘father’ or ‘mother’ of invention.

Muybridge was born in 1830 as Edward Muggeridge, into a merchant family that traded in corn and coal in Kingston upon Thames in England. The place inspired the first of Muybridge’s many name changes when, at 20, he appropriated the ‘Eadweard’ spelling of the Anglo-Saxon kings that had been carved upon an ancient coronation stone near his home. He set off to New York as a young man, in 1850, before crossing the country to the frontier town of San Francisco. Over time, for reasons he never explained, his surname evolved to Muygridge and then Muybridge.

Muybridge set up as a professional photographer and, in 1872, he married Flora Shallcross Stone, a young divorcee half his age. He also found himself drawn into Stanford’s circle, after the millionaire asked Muybridge to take photographs of his galloping racehorses to determine whether they had all four hooves off the ground at any point in their stride.

With Muybridge away from home much of the time, Flora fell pregnant to a rambunctious drama critic called Harry Larkyns. Seven months after the baby was born, Muybridge discovered he was not the father. Incensed, he tracked down Larkyns at a ranch in the Napa Valley. Muybridge called out from his hiding place in the dark. As his wife’s lover peered into the gloom, Muybridge said: ‘My name is Muybridge and I have a message from my wife,’ shooting Larkyns point-blank through the heart. Within hours, Muybridge had been arrested.

Charles Babbage led a much more refined life than the knockabout Muybridge. Born in London in 1791, son of a goldsmith and banker, Babbage inherited a fortune from his father and could have spent his life as a dilettante. He thrived in London’s high society, and much of his work seems to have been undertaken in an attempt to impress the rich and famous at his popular soirées. But Babbage was also intelligent and well-educated, holding down the post as Lucasian Professor of Mathematics at Cambridge for 11 years from the age of 37.

Early on in his tenure, inspired by the industrial revolution, he began toying with the idea of a mechanical calculator that would use gears to overcome the labour of working out mathematical equations by hand. In the summer of 1821, Babbage was helping his astronomer friend John Herschel check a series of astronomical tables. Going cross-eyed with the effort of working the array of figures, Babbage is said to have cried out: ‘My God, Herschel! How I wish these calculations could be executed by steam!’

Of itself, the idea of a mechanical calculator was not new. Such devices go back at least as far as the Antikythera mechanism, recovered from a Greek shipwreck dating to the first or second century BC, which used a complex mechanism of gears to predict the motion of heavenly bodies and other natural phenomena. And there is a more direct antecedent of Babbage’s work in the calculating machine devised by the French mathematician Blaise Pascal in the 1640s, a number of which were constructed.

Babbage’s first concept was called a ‘Difference Engine’. Like Pascal’s machines, it involved a series of gears, but was more sophisticated in the range and scale of its calculations. He convinced the British government to invest £17,000 in his project – around £1.2 million in today’s money – but he completed only a fraction of the total machine. Despite the government’s objections, he dropped the Difference Engine for a far grander idea – what he called his ‘Analytical Engine’.

Muybridge’s breakthrough came with the zoopraxiscope, the world’s first movie projector

Muybridge’s murder trial in 1875 drew a huge crowd. His defence team attempted to show that he was deranged, arguing that a wagon crash in 1860 had damaged his judgment and self-control. But the prosecution tore his insanity plea apart. In a final, impassioned speech, Muybridge’s defence lawyer told the jury that Muybridge’s actions were justified, citing the Bible to argue that killing his wife’s adulterous lover was the right thing to do. After a night’s deliberation, the jury found Muybridge not guilty.

Some time after, Muybridge reconnected with Stanford. This time, he set up a bank of 12 top-quality stereoscopic cameras with high-speed shutters, and took a rapid series of photographs of Stanford’s horse in motion. The breakthrough came with the invention of the zoopraxiscope, the device that had enabled Stanford to recognise his horse. Images were arrayed around the outside of a disc, which rotated rapidly in one direction, while a counter-rotating disc with slots acted as a gate to control which image was projected onto a screen, creating the illusion of movement. It was the world’s first movie projector.

After a sell-out European tour and a legal battle with Stanford, who claimed the images as his own, Muybridge found another opportunity to raise his profile. He met William Pepper, the provost of the University of Pennsylvania, who enabled Muybridge to produce thousands of motion studies. Between 1884 and 1887, using far better photographic materials, Muybridge shot hundreds of sequences of men and women, often naked, performing all sorts of tasks and movements. (It was often very difficult to persuade bricklayers to do their job with no clothes on, Muybridge commented ruefully.) The apex of his contribution to moving pictures came at the World’s Columbian Exposition of 1893, a huge fair in Chicago to mark the 400th anniversary of Christopher Columbus landing in the New World. Here, Muybridge built the Zoopraxographical Hall – the first purpose-built cinema, a 50-foot-high extravaganza in mock stone.

In contrast to Muybridge’s raw and dusty work on Stanford’s property, Babbage was inspired by the sophistication of silk weaving. Making complex patterns with fine silk thread was painfully slow when done by hand – so much so that two loom operators might produce only an inch of material a day. In the 1740s, a French factory inspector devised a loom that used a mechanism such as a musical box to speed up the process. Just as the pins on the rotating cylinder of a musical box triggered notes on metal prongs, the device used pins to control different-coloured threads. However, each cylinder was expensive to produce, and the design was limited by the size of the cylinder – one turn, and the pattern began to repeat.

A new loom created by Joseph-Marie Jacquard, the son of a master weaver, swapped the cylinder for a series of holes punched on cards. Each hole indicated whether or not a particular colour should be used at that point, and because the train of punched cards could be as long as the pattern required, almost any piece of weaving could be automated this way. Before long, Jacquard looms were turning out two feet of silk a day – a remarkable transformation of productivity.

The versatility of Jaquard’s system appealed to Babbage. A treasure he often exhibited to visitors was a portrait of Jacquard that appeared to be an etching – but on close examination it was woven from silk, with a remarkable 24,000 rows of thread making up the image. Such a product would have been impossible without Jacquard’s technology, and Babbage realised that a similar approach could be used in a truly revolutionary computing device, his Analytical Engine.

Dismissing the fixed gears of his earlier design, Babbage wanted the Analytical Engine to have the same flexibility as the Jacquard loom. For the Difference Engine, the data to be worked on was to be entered manually on dials, with the calculation performed according to the configuration of gears. In the Analytical Engine, both data and calculation would be described by a series of Jacquard-style punched cards, which allowed for far more flexibility of computation.

The Difference Engine was an incomplete mechanical calculator, while the Analytical Engine never got off the drawing board

There was just one problem. Although Babbage designed the Analytical Engine in concept, he never managed to construct even a part of it. Indeed, it’s unlikely that his design could ever have been successfully built. His plans fired the enthusiasm of Ada Lovelace, the mathematician and daughter of the poet Lord Byron. She was eager to work with Babbage on his Analytical Engine, and described several potential programs for his hypothetical machine. But Babbage showed little interest in Lovelace’s contributions. His grand vision proved impossible to make a reality.

The technologies envisaged by both Babbage and Muybridge bear little connection to their modern equivalents. Their devices were evolutionary dead ends. Muybridge’s banks of cameras were clumsy and impractical; his movies were limited to a couple of seconds in duration. And Babbage’s computers were even worse. The Difference Engine was an incomplete mechanical calculator, while the Analytical Engine never got off the drawing board.

For both computers and moving pictures, the real, usable technology would require a totally different approach. But how the legacies of each of these inventors has been preserved varied greatly according to the vagaries of chance, politics and ambition.

The conceptual originators of the modern computer were the British mathematicians Alan Turing, who devised the fundamental model, and John von Neumann, who turned Turing’s highly stylised theory into a practical architecture. These pioneers had an academic background, and their consideration was not glory, but solving an intellectual challenge to help the military effort during the Second World War. As prime minister, Winston Churchill was determined to keep the power of the computing equipment at Britain’s disposal a secret, and the work was accordingly downplayed. As a result, Babbage was never totally eclipsed by his successors.

But neither academic restraint nor political interference got in the way of Muybridge’s rivals. Inventors such as the Lumières, two French brothers who developed a self-contained camera and movie projector, had everything to gain financially from being recognised as firsts. They picked up on the invention of the roll-film for still cameras to create moving pictures that were much easier to make, and lasted much longer. These entrepreneurs had no need for a conceptual ancestor – Muybridge was not a muse, but potential competition.

Muybridge’s reputation also suffered after the publication of A Million and One Nights (1926), a book about the early years of moving pictures by Terry Ramsaye, the editor of an American cinema trade magazine. Ramsaye cast Muybridge as a self-serving fraud who passed off other people’s inventions as his own. He was supported by the evidence of John D Isaacs, an electrical engineer who helped to build the shutter-release mechanism used in Muybridge’s action photography, and claimed to be the genius behind all Muybridge’s work. Muybridge, who had died more than 20 years earlier, couldn’t speak for himself. Ramsaye’s account was later discredited, but it was enough to wipe Muybridge off the map for many years.

Science and technology are rarely about lone genius. Neither Muybridge nor Babbage developed workable inventions that functioned at scale, and it ultimately took new creators, with fresh approaches, to bring their ideas to life. But that initial spark matters – and, as their lives remind us, being an inventor is as much about imagination as it is about creation.

How can a first-person shooter have a victim complex?

 A lot of terrible things happen to video-game characters. In the early days of the form, Italian plumbers were squashed by barrels, loveable hedgehogs impaled on spikes, and heroic astronauts exploded. But no game delights in inflicting as much horror on its protagonists as the $10-billion Call of Duty (CoD) franchise.

Originally set in the Second World War, the series took on its current ultra-popular form with Call of Duty 4: Modern Warfare (2007). Every year brings new CoD games, and the franchise regularly tops charts, racking up more than 175 million sales over the course of the past decade. Each game weaves together a semi-coherent narrative that combines the regular annihilation of its characters at a George R R Martin pace with a wider and more worrying story arc: one of US powerlessness, decline and revenge.

The Call of Duty games are first-person shooters (FPS). ‘First-person’ means that the players see the world through the eyes of their characters – only their hands and weapons are clearly visible. (Some games break this form in third-person cutscenes, but CoD sticks to it resolutely). ‘Shooter’ means that the character interacts with the world almost entirely through the barrel of a gun; the environment, and its endless stream of enemies, exists to be destroyed.

Many FPS are power fantasies, with the child-like joy of being able to murder everything you see. These are little boys’ tales of war, where a finger-gun blows away a thousand foes, and the player is always the actor, never the victim.

Call of Duty frequently reverses that dynamic. In almost every other FPS, the player’s death is a constant possibility, yet those are transient endings, erased from the game’s reality by a simple reload. But in Call of Duty, sufferings both permanent and unavoidable are ineffably worked into the game’s narrative. Although the gameplay is a consistent frenzy of violence inflicted by the player on the world, narrative control – the happy freedom to move, shoot, call in drone strikes, knife people in the back – is regularly snatched back from the players, forcing them into positions of constant helplessness.

Meanwhile, the point of view continuously switches, deliberately and disorientatingly – a CIA agent, a doomed ISS astronaut, a fallen dictator, a SAS operative. Throughout the series, the player’s avatars are repeatedly tortured, nuked, brainwashed, murdered, mutilated and, above all, betrayed – often by their own leadership. If they live, they are depicted as broken, made whole only by vengeance. If they die, another character takes up the mantle of revenge.

Beyond such personal treacheries and dismemberments is a wider depiction of victimhood, not individual but national. The reversal, from active shooter to passive martyr, gets at a lasting psychological truth, which is that however absolute US military superiority is in reality, many Americans feel themselves to be a nation under siege by an ungrateful world, eternally vulnerable.

The original CoD 4: Modern Warfare (2007) begins by depicting US power realistically (enemy tanks are neutralised by man-portable FGM-148 Javelin missiles, enemy soldiers are mowed down en masse by AC-130 Spectre gunships, and the war is far from US shores). Then a nuclear blast kills the first protagonist, shifting the conditions of the game. Alongside him die 30,000 other US troops, their names and ranks scrolling past rapidly on screen, in a plot that turns out to have been orchestrated by a line-up of villains old and new – Russian ultra-nationalists working alongside Islamic terrorists.

The sequels go further. Through a technological magical wand that cripples US defence systems, an army of Russian paratroopers and marines is able to invade Virginia. In the course of MW2 (2009) and MW3 (2011), battles rage along Wall Street, the Brooklyn Bridge and Pennsylvania Avenue, with ‘tens of thousands’ of Americans killed and the day saved only by heroic violence from the US army.

In this timeline, US power is expressly fallen. As the ad copy for Call of Duty: Ghosts (2013) puts it, ‘the balance of global power’ changed forever and a ‘crippled nation’ faces ‘technologically superior’ foes. War, against an array of bogeymen, from insurgent South American powers to the ever-green Russians, has become a constant necessity.

Every indignity inflicted on individual Americans (and the occasional Brit) in the narrative is echoed in the games’ geopolitics: US cities burn in nuclear fire, Mexican border states become a ‘No Man’s Land’, and the US financial and military elite repeatedly betray the nation, and its soldiers.

This might all seem very silly, and it is. I don’t believe that the writers of the game have much more in mind than spectacle when plotting the course of the series, because it certainly does look pretty when great landmarks explode. Token efforts at moral complexity are woven in here and there, from the anti-war quotes that play over every temporary death to the origins of a foe’s plot against the US in the death of his relatives from a US strike.

And yet, Modern Warfare reinforces, consciously or otherwise, the ever-present US myth that the country is an innocent victim in a cruel world. It’s a belief that swelled massively after the attacks of 11 September 2001, but it has always been present. The start of US wars has always been framed by betrayal, real or otherwise, from the actual surprise attacks of Pearl Harbor to the imaginary Spanish super-weapons blamed for the explosion of the USS Maine in 1898 or US President Lyndon B Johnson’s lies around the Gulf of Tonkin. And in Modern Warfare, the killing of US soldiers, especially by illegitimate foes – militias, terrorists, ‘rebel forces’ that make up the mass of Call of Duty’s shooting targets – is taken not just as a consequence of war, but as a crime committed by the enemy.

Coupled with that is the fear of weakness. In a country that outspends all its potential foes put together, nearly half the public believes, according to a recent Gallup poll, that it is ‘just one of several leading military powers’. (Back in the Cold War, the entirely fictitious ‘missile gap’ served the same propagandist purpose, but then, at least, there was the excuse of Soviet opaqueness around what were, in fact, considerably smaller and more backward arsenals than the US possessed.)

Victimhood is only the start. The protagonists of CoD games aren’t stopped by betrayal, injury or even death. Their travails are the necessary prelude to their roaring rampage of vengeance, and their passive suffering doesn’t subvert the fantasy of power but endorses it; every bullet fired is justified by ruined bodies, both politically and personally. CoD’s wars are a hysterically exaggerated reinforcement of national wish-fulfilment, consumed by millions of young US men: they hurt us first, so we get to hurt them back.

We have greater moral obligations to robots than to humans

 Down goes HotBot 4b into the volcano. The year is 2050 or 2150, and artificial intelligence has advanced sufficiently that such robots can be built with human-grade intelligence, creativity and desires. HotBot will now perish on this scientific mission. Does it have rights? In commanding it to go down, have we done something morally wrong?

The moral status of robots is a frequent theme in science fiction, back at least to Isaac Asimov’s robot stories, and the consensus is clear: if someday we manage to create robots that have mental lives similar to ours, with human-like plans, desires and a sense of self, including the capacity for joy and suffering, then those robots deserve moral consideration similar to that accorded to natural human beings. Philosophers and researchers on artificial intelligence who have written about this issue generally agree.

I want to challenge this consensus, but not in the way you might predict. I think that, if we someday create robots with human-like cognitive and emotional capacities, we owe them more moral consideration than we would normally owe to otherwise similar human beings.

Here’s why: we will have been their creators and designers. We are thus directly responsible both for their existence and for their happy or unhappy state. If a robot needlessly suffers or fails to reach its developmental potential, it will be in substantial part because of our failure – a failure in our creation, design or nurturance of it. Our moral relation to robots will more closely resemble the relation that parents have to their children, or that gods have to the beings they create, than the relationship between human strangers.

In a way, this is no more than equality. If I create a situation that puts other people at risk – for example, if I destroy their crops to build an airfield – then I have a moral obligation to compensate them, greater than my obligation to people with whom I have no causal connection. If we create genuinely conscious robots, we are deeply causally connected to them, and so substantially responsible for their welfare. That is the root of our special obligation.

Frankenstein’s monster says to his creator, Victor Frankenstein:

I am thy creature, and I will be even mild and docile to my natural lord and king, if thou wilt also perform thy part, the which thou owest me. Oh, Frankenstein, be not equitable to every other, and trample upon me alone, to whom thy justice, and even thy clemency and affection, is most due. Remember that I am thy creature: I ought to be thy Adam….

We must either only create robots sufficiently simple that we know them not to merit moral consideration – as with all existing robots today – or we ought to bring them into existence only carefully and solicitously.

Alongside this duty to be solicitous comes another, of knowledge – a duty to know which of our creations are genuinely conscious. Which of them have real streams of subjective experience, and are capable of joy and suffering, or of cognitive achievements such as creativity and a sense of self? Without such knowledge, we won’t know what obligations we have to our creations.

Yet how can we acquire the relevant knowledge? How does one distinguish, for instance, between a genuine stream of emotional experience and simulated emotions in an artificial mind? Merely programming a superficial simulation of emotion isn’t enough. If I put a standard computer processor manufactured in 2015 into a toy dinosaur and program it to say ‘Ow!’ when I press its off switch, I haven’t created a robot capable of suffering. But exactly what kind of processing and complexity is necessary to give rise to genuine human-like consciousness? On some views – John Searle’s, for example – consciousness might not be possible in any programmed entity; it might require a structure biologically similar to the human brain. Other views are much more liberal about the conditions sufficient for robot consciousness. The scientific study of consciousness is still in its infancy. The issue remains wide open.

If we continue to develop sophisticated forms of artificial intelligence, we have a moral obligation to improve our understanding of the conditions under which artificial consciousness might genuinely emerge. Otherwise we risk moral catastrophe – either the catastrophe of sacrificing our interests for beings that don’t deserve moral consideration because they experience happiness and suffering only falsely, or the catastrophe of failing to recognise robot suffering, and so unintentionally committing atrocities tantamount to slavery and murder against beings to whom we have an almost parental obligation of care.

We have, then, a direct moral obligation to treat our creations with an acknowledgement of our special responsibility for their joy, suffering, thoughtfulness and creative potential. But we also have an epistemic obligation to learn enough about the material and functional bases of joy, suffering, thoughtfulness and creativity to know when and whether our potential future creations deserve our moral concern.

The info moralist

 Persecuted little guy, or powerful revolutionary – what sort of wunderkind was Aaron Swartz?

Who was Aaron Swartz? I never met him, though I’ve had dealings with friends of his over the years. The outline of his biography is a matter of public record: teenaged computer whizz gets rich, becomes a political activist and ends up in his 20s facing decades in jail for murky charges related to the misappropriation of academic journal articles. That much is on Wikipedia.

If that isn’t intimate enough, perhaps his character comes through in the tributes that poured onto the internet following Swartz’s suicide in 2013. The signature notes of tenderness, exasperation and awe, in reminiscences from Tim Berners-Lee, Lawrence Lessig, Cory Doctorow and many other notable mentors, certainly conjure a fleeting presence. Nevertheless, in the end, the person is irrecoverable, and those of us who weren’t lucky enough to know him never will.

‘What was Aaron Swartz?’, on the other hand, seems like both a tractable and a worthwhile question, not least because a decent answer ought to say something about where we are now. Swartz positioned himself at the exact spot where technology and politics press noses and glare at one another. It’s a Silicon Valley joke (or perhaps just a Silicon Valley joke) that every idiot with a dating app says he wants to change the world, but Swartz seems really to have meant it. He quit money the way PayPal’s co-founder Peter Thiel wants smart kids to quit college. He became a white-hat hacker among the levers of state power.

And things ended, not just badly, but dismally, in a sulphurous halfworld of G-men, prosecutorial intimidation and forced betrayals. It is, I suspect, impossible to learn anything about the young activist’s story without starting to see it as a symbol of something ominous in our present chunk of history. But what?

An all-round prodigy raised among computer enthusiasts, at the age of 13 Swartz created a website called ‘The Info Network’, an encyclopedia designed to be written and edited by its users. This was in 1999: two years before Wikipedia. The Info Network came to nothing, as did Swartz’s petition site, watchdog.net, a proto-version of change.org that he cooked up around the same time. But another project he co-authored did rather better: RSS became the standard format for online feeds (this website uses it and, if you have a site of your own, there is a good chance that yours does too). Then, in 10th grade, Swartz dropped out of high school.

While dabbling in a few different college courses around his family home in Chicago, he helped to decide the terms for Creative Commons content licences, the standard agreements under which creative works are shared on the internet. He did a stint at Paul Graham’s start-up incubator, Y Combinator, where he joined the team that founded Reddit, a commenting platform that came to be known as ‘the front page of the internet’. That site got bought out by the publisher Condé Nast, giving Swartz a sack of money and an unhappy few weeks drifting around the offices of Wired magazine. Any other young celebrity founder might have tried to repeat his entrepreneurial coup. By now, though, Swartz was interested in something bigger than mere success.

He helped to set up the Progressive Change Campaign Committee, a research organisation designed to steer US politics in a left-Democrat direction. Another group he co-founded in 2010, Demand Progress, became the centre of the campaign against the Stop Online Piracy Act (SOPA), a heavy‑handed attempt to prevent illegal file‑sharing by shutting down websites. Swartz was writing copiously on his Raw Thought blog: political screeds, tough-minded cultural commentary, reading lists, curious jokes. He also put his name to the ‘Guerilla Open Access Manifesto’, as pithy a statement as exists of his core political philosophy:

Information is power. But like all power, there are those who want to keep it for themselves. The world’s entire scientific and cultural heritage, published over centuries in books and journals, is increasingly being digitised and locked up by a handful of private corporations… We can fight back. Those with access to these resources – students, librarians, scientists – you have been given a privilege. You get to feed at this banquet of knowledge while the rest of the world is locked out. But you need not – indeed, morally, you cannot – keep this privilege for yourselves. You have a duty to share it with the world.

Perhaps it was in the service of this ideal that Swartz began a series of projects involving massive downloads of scholarly and legal archives. He created a script that worked through thousands of law review articles to determine whether corporate sponsorship exerted any influence on scholarly conclusions (guess). In 2008 he downloaded 2.7 million pages of federal court documents from the US Government’s PACER database and released them online. Why not? The documents were public domain anyway; the fact that the government billed you eight cents a page if you wanted to access them was rent-farming of the most obvious kind. The FBI investigated him for that, but no charges emerged. That round went to Swartz. But then there was JSTOR.

JSTOR (for Journal Storage) is an online repository of scholarly articles. It charges for access but – and this is the bit that sticks in the craw of open-information activists – contributes nothing to the content it hosts, which is the work of uncompensated academics. Nevertheless, the content is not public domain: copyright usually rests with the publishers.

he was smart: the kind of patient, brutally practical intelligence that actually accomplishes things

In the autumn of 2010, Swartz snuck into the Massachusetts Institute of Technology (MIT), a university of which he was not a member (presumably he would have had an easier time at Harvard, where he held a research fellowship), and set up a laptop to download as much of JSTOR as he could grab. Someone found the computer in a service closet and, oddly, set up a spycam to record its owner on his return. Swartz had to go back to switch a hard drive. And so he was caught on camera, and a little later, in the flesh.

It was never clear exactly what he intended to do with the documents he had obtained. On the face of it, his biggest crimes were trespassing and violation of MIT’s computer policy. Yet somewhere in the recesses of the law-enforcement apparatus, it appears Swartz’s card was marked. He learned that he faced charges of breaking and entering with intent to commit a felony; projected sentence: 35 years. MIT tried to wash its hands of the affair, but the prosecutor was implacable. The legal defence swallowed his Reddit money. Federal agents intimidated his loved ones.

Swartz had suffered from depression at intervals throughout his life. He seems, for example, to have considered suicide shortly after he left Wired, writing a story on his blog about a person called Aaron who kills himself after losing a job. The story is disconcerting, but Swartz freely admitted to an angsty streak in his creative writing, once referring derisively to his own ‘old tortured-psyche fiction pieces’. However, in 2013, at the age of 26, he actually went through with it. It was no longer possible to know Aaron Swartz.

‘Growing up, I slowly had this process of realising,’ Swartz announces, gazing at the viewer with hypnotic self-assurance, ‘that all the things around me that people had told me were just the natural way things were… weren’t natural at all. They were things that could be changed. And they were things that, more importantly, were wrong and should change.’

This line, quoted close to the beginning of The Internet’s Own Boy (2014), Brian Knappenberger’s remarkable documentary about Swartz’s life, seems to contain our first key to the meaning of the whole. At first sight, it just sounds like ordinary youthful idealism. If we wanted to peg it on any specific political temperament, perhaps there’s an echo of old-style Fabianism – think George Bernard Shaw’s ‘When will we realise that the fact that we can become accustomed to anything… makes it necessary to examine carefully everything we have become accustomed to?’ And to a certain extent, the cap fits: Swartz surely does sit in that muscular reformist tradition.

But look at his autobiographical statement again. What is the implied sequence of discoveries? First, that things could be changed, and only then that they should be changed. Can precedes ought, not merely in the logical sense, but developmentally. I suspect that, for Swartz, this was really how it happened. He wasn’t, in the first instance, a dreamer who sought the tools he might need; he was a technologist who noticed some affordances and began to plot a course. He might have been naïve in various ways, but he wasn’t wishful.

‘The trick,’ he once wrote on the general theme of ambitious projects, ‘is to set yourself lots of small challenges along the way. If your start‑up is eventually going to make a million dollars, can it start by making 10? If your book is going to eventually persuade the world, can you start by persuading your friends? Instead of pushing all your tests for success way off to the indefinite future, see if you can pass a very small one right now.’ In short, I think Swartz is best understood as a very driven and ambitious sort of engineer.

Knappenberger’s film for the most part paints him as two rather different, and rather more familiar, sorts of protagonist. The first is the little guy, broken by an inhuman (or regrettably human) system. The second is the implacable revolutionary, boldly facing the future. Neither characterisation seems false, exactly, but they both miss the important thing.

Take the first. For the supposed crime of downloading academic papers without permission, Swartz really was facing 35 years in jail and $1 million in fines. He was an emotionally fragile 26-year-old with powerful enemies. The FBI was cruising around his neighbourhood, scaring him silly. The prosecutor wanted to make an example of him. Poor kid, right?

Well, yes. But at the same time, he was a good deal more powerful than that picture suggests. He had a comfortable cushion of money. His development work on RSS had given him enormous social capital among alpha nerds. Other influential friends opened doors for him in law, politics, the media. He was – you can see it in the extensive video interviews – intensely charismatic. And most enviably, he was smart: the kind of patient, brutally practical intelligence that actually accomplishes things (and this is to say nothing of his programming abilities). If it wasn’t for the precariousness of his mental state, it’s easy to imagine him beating the charges against him. Smaller Davids have beaten their Goliaths.

If not quite the little guy, then was he a revolutionary? He certainly looked like one, with his scrubby Che Guevara beard and plumes of hair. But if he was, it was of a very particular, rather technically minded sort – a far cry from the sacrificial lambs of the Arab Spring or Occupy. If he was dreaming of a better world, he was determined not to get carried away. (‘Can you start by persuading your friends?’) Swartz believed in crowds, but never leaderlessness. In fact he gave a good deal of thought to the dilemmas of command, examining, for example, the way in which companies, ‘even as they get big… betray facets of the founder’s personality’. ‘An organisation,’ he wrote in one of his final blog posts, ‘is not just a pile of people, it’s also a set of structures. It’s almost like a machine made of men and women.’ And those machines needed an intelligent engineer to make them work.

the world is changing anyway. Swartz was just one of the people who wanted to steer it

The political strategist Matt Stoller said that his friend Swartz approached politics just like he approached technology: ‘His method was as follows – (1) Learn (2) Try (3) Gab (4) Build.’ The pair hung around Congress, talking to lobbyists and policymakers. Swartz was learning the processes and the language, getting his head round the system. It was around this time that he started to talk (tongue not audibly in cheek) like the true heir to the spirit of the republic. Explaining to a TV reporter why he wanted to block legislation that would allow the government to shut down any website that hosted pirated content, he declared: ‘The principle is one that I think our founding fathers would have understood, if the internet had been around back then.’ Addressing a crowd on the same subject, he announces that SOPA would mean that ‘The freedoms guaranteed in our Constitution, the freedoms our country had been built on, would be suddenly deleted.’ Swartz was getting into character.

All the same, it’s strange to hear him presenting himself as a rediscoverer of old verities and an exposer of old lies. In reality, his domain was a sphere of near-total novelty. He was one of the people who found, or placed, a deep moral significance in the web. At stake in the battles over SOPA was ‘the right to connect’. Is there any such right? Since when? Listening to Swartz, you have the sense of a new moral order being conjured out of the confusion of the present. He was laying claims on the virgin territory of the internet, making space for concerns beyond the inevitable machinations of money and power. The naive activist wants to change the world. But that isn’t necessary: the world is changing anyway. Swartz was just one of the people who wanted to steer it.

In this, he seems much less like those sweet and hopeless Occupiers and more like one of the enigmatic, entrepreneurial operators who loom over our information politics. Edward Snowden is the obvious one; Julian Assange, too – men for whom ideology and opportunity seem inseparable. Add the (possibly pseudonymous) inventor of Bitcoin, Satoshi Nakamoto, to this list. Another who springs to mind is Mark Zuckerberg, whose claim that ‘Facebook… was built to accomplish a social mission – to make the world more open and connected’ reads as a kind of plasticky corporate take on the Guerilla Open Access Manifesto. In every case, the engineer has started to operate as a visionary improviser, seeing an adjacent world-state within the world system and instantly imbuing it with the radioactive glow of moral mission.

I admire them all, in different ways. Perhaps we need them. It can be difficult sometimes to see the internet as a collection of contingent ideas: it appears to unfold with a revolutionary logic of its own, so that the personalities of its vanguard party dissolve in the onrushing spirit of the age.

Perhaps, though, a version of what Swartz called Founder’s Syndrome applies, and the world we live in does after all ‘betray facets of the founder’s personality’, not only in the forms of our infrastructure but in the moral sense that we use to understand it all. In my list of great boy wonders, Swartz seems the most prepossessing of the lot, most animated by a consistent and recognisable politics. If nothing else, his moralising drew on a decent reading list. But as our lives are dominated ever more completely by complex computer systems, it is a little disquieting to realise that perhaps our heroes must be as alien and inscrutable as our problems.

The attention economy

 It costs nothing to click, respond and retweet. But what price do we pay in our relationships and our peace of mind?

How many other things are you doing right now while you’re reading this piece? Are you also checking your email, glancing at your Twitter feed, and updating your Facebook page? What five years ago David Foster Wallace labelled ‘Total Noise’ — ‘the seething static of every particular thing and experience, and one’s total freedom of infinite choice about what to choose to attend to’ — is today just part of the texture of living on a planet that will, by next year, boast one mobile phone for each of its seven billion inhabitants. We are all amateur attention economists, hoarding and bartering our moments — or watching them slip away down the cracks of a thousand YouTube clips.

If you’re using a free online service, the adage goes, you are the product. It’s an arresting line, but one that deserves putting more precisely: it’s not you, but your behavioural data and the quantifiable facts of your engagement that are constantly blended for sale, with the aggregate of every single interaction (yours included) becoming a mechanism for ever-more-finely tuning the business of attracting and retaining users.

Consider the confessional slide show released in December 2012 by Upworthy, the ‘website for viral content’, which detailed the mechanics of its online attention-seeking. To be truly viral, they note, content needs to make people want to click on it and share it with others who will also click and share. This means selecting stuff with instant appeal — and then precisely calibrating the summary text, headline, excerpt, image and tweet that will spread it. This in turn means producing at least 25 different versions of your material, testing the best ones, and being prepared to constantly tweak every aspect of your site. To play the odds, you also need to publish content constantly, in quantity, to maximise the likelihood of a hit — while keeping one eye glued to Facebook. That’s how Upworthy got its most viral hit ever, under the headline ‘Bully Calls News Anchor Fat, News Anchor Destroys Him On Live TV’, with more than 800,000 Facebook likes and 11 million views on YouTube.

But even Upworthy’s efforts pale into insignificance compared with the algorithmic might of sites such as Yahoo! — which, according to the American author and marketer Ryan Holiday, tests more than 45,000 combinations of headlines and images every five minutes on its home page. Much as corporations incrementally improve the taste, texture and sheer enticement of food and drink by measuring how hard it is to stop eating and drinking them, the actions of every individual online are fed back into measures where more inexorably means better: more readers, more viewers, more exposure, more influence, more ads, more opportunities to unfurl the integrated apparatus of gathering and selling data.

Attention, thus conceived, is an inert and finite resource, like oil or gold: a tradable asset that the wise manipulator auctions off to the highest bidder, or speculates upon to lucrative effect. There has even been talk of the world reaching ‘peak attention’, by analogy to peak oil production, meaning the moment at which there is no more spare attention left to spend.

This is one way of conceiving of our time. But it’s also a quantification that tramples across other, qualitative questions — a fact that the American author Michael H Goldhaber recognised some years ago, in a piece for Wired magazine called ‘Attention Shoppers!’ (1997). Attention, he argued, ‘comes in many forms: love, recognition, heeding, obedience, thoughtfulness, caring, praising, watching over, attending to one’s desires, aiding, advising, critical appraisal, assistance in developing new skills, et cetera. An army sergeant ordering troops doesn’t want the kind of attention Madonna seeks. And neither desires the sort I do as I write this.’

For all the sophistication of a world in which most of our waking hours are spent consuming or interacting with media, we have scarcely advanced in our understanding of what attention means. What are we actually talking about when we base both business and mental models on a ‘resource’ that, to all intents and purposes, is fabricated from scratch every time a new way of measuring it comes along?

In Latin, the verb attendere — from which our word ‘attention’ derives — literally means to stretch towards. A compound of ad (‘towards’) and tendere (‘to stretch’), it invokes an archetypal image: one person bending towards another in order to attend to them, both physically and mentally.

Attending is closely connected to anticipation. Soldiers snap to attention to signify readiness and respect — and to embody it. Unable to read each others’ minds, we demand outward shows of mental engagement. Teachers shout ‘Pay attention!’ at slumped students whose thoughts have meandered, calling them back to the place they’re in. Time, presence and physical attentiveness are our most basic proxies for something ultimately unprovable: that we are understood.

The best teachers, one hopes, don’t shout at their students — because they are skilled at wooing as well as demanding the best efforts of others. For the ancient Greeks and Romans, this wooing was a sufficiently fine art in itself to be the central focus of education. As the manual on classical rhetoric Rhetorica ad Herennium put it 2,100 years ago: ‘We wish to have our hearer receptive, well-disposed, and attentive (docilem, benivolum, attentum).’ To be civilised was to speak persuasively about the things that mattered: law and custom, loyalty and justice.

This vision of puppeteers effortlessly pulling everyone else’s strings — however much it might fulfil both geek fantasies and Luddite nightmares — is distinctly dubious

Underpinning this was neither honour nor idealism, but pragmatism embodied in a five-part process. Come up with a compelling proposition, arrange its elements in elegant sequence, polish your style, commit the result to memory or media, then pitch your delivery for maximum impact. Short of an ancient ‘share’ button, the similarities to Upworthy’s recipe for going viral are impressive. Cicero, to whom Rhetorica ad Herennium is traditionally attributed, also counted flattery, bribery, favour-bargaining and outright untruth among the tools of his trade. What mattered was results.

However, when it comes to automated systems for garnering attention, there’s more at play than one person listening to another; and the processes of measurement and persuasion have some uncannily totalising tendencies. As far as getting the world to pay attention to me online, either I play by the rules of the system — likes, links, comments, clicks, shares, retweets — or I become ineligible for any of its glittering prizes. As the American writer and software engineer David Auerbach put it in n+1 magazine, in a piece pointedly titled ‘The Stupidity of Computers’ (2012), what is on screen demands nothing so much as my complicity in its assumptions:

Because computers cannot come to us and meet us in our world, we must continue to adjust our world and bring ourselves to them. We will define and regiment our lives, including our social lives and our perceptions of our selves, in ways that are conducive to what a computer can ‘understand’. Their dumbness will become ours.

In computing terms, to do things in a way the system does not ‘understand’ is to do nothing at all. It is to be incomprehensible, absurd, like trying to feed a banana instead of paper into a printer. What counts is synonymous with what’s being counted.

All of which seems to place immense power, not to mention responsibility, into the hands of the system architects: the coders, designers, advertisers, professional media manipulators and social media gurus devoted to profitable clicking.

Yet this vision of puppeteers effortlessly pulling everyone else’s strings — however much it might fulfil both geek fantasies and Luddite nightmares — is distinctly dubious. As the British economist Charles Goodhart argued in 1975 in an aphorism that has come to be known as Goodhart’s law, ‘When a measure becomes a target, it ceases to be a good measure.’ There are few better summaries of the central flaw in attention economics. Attention-engineers are effectively distributing printing presses for a private currency — and with everyone else desperate to churn out as much as possible, by any means possible, what’s going on is more a chaotic scramble for advantage than a rational trade in resources.

No matter how cunning the algorithms and filters, entire industries of manufactured attention bloom and fade around every possibility of profit. As recent investigations have suggested, achievements in the field range from ‘click farms’ of low-paid workers churning out ersatz engagement to paid endorsements from social media celebrities, via bulk-purchased followers and fake grassroots activists. Every target is continually being moved, refined and undermined. Nobody is in control.

And who is to say that they should be? Seeing data writ large, relations spelt out and chains of consequence snaked brightly across the recorded realm, we confuse information with mastery. Yet this is at best a category error, and at worst a submission to wishful bullshit: a mix of convenient propaganda and comforting self-deception that hails new kinds of agency, without pausing to acknowledge the speciousness of much of what’s on offer.

In the preface to his essay collection Tremendous Trifles (1909), the English author, ontologist and professional paradox-weaver G K Chesterton told the fable of two boys who were each granted a wish. One chose to become a giant, and one to become extremely small. The giant, to his surprise, found himself bored by the shrunken land beneath him. The tiny boy, however, set off gladly to explore the endless world of wonders his front garden had become. The moral, as Chesterton saw it, was one of perspective:

If anyone says that I am making mountains out of molehills, I confess with pride that it is so. I can imagine no more successful and productive form of manufacture than that of making mountains out of molehills… I have my doubts about all this real value in mountaineering, in getting to the top of everything and overlooking everything. Satan was the most celebrated of Alpine guides, when he took Jesus to the top of an exceeding high mountain and showed him all the kingdoms of the earth. But the joy of Satan in standing on a peak is not a joy in largeness, but a joy in beholding smallness, in the fact that all men look like insects at his feet.

There’s a similarly reductive exaltation in defining attention as the contents of a global reservoir, slopping interchangeably between the brains of every human being alive. Where is the space, here, for the idea of attention as a mutual construction more akin to empathy than budgetary expenditure — or for those unregistered moments in which we attend to ourselves, to the space around us, or to nothing at all?

If contentment and a sense of control are partial measures of success, many of us are selling ourselves far too cheap

From the loftiest perspective of all, information itself is pulling the strings: free-ranging memes whose ‘purposes’ are pure self-propagation, and whose frantic evolution outstrips all retrospective accounts. This is the mountaintop view of Chesterton’s Satan, whispering in a browser’s ear: consider yourself as interchangeable as the button you’re clicking, as automated as the systems in which you’re implicated. Seen from such a height, you signify nothing beyond your recorded actions.

Like all totalising visions, it’s at once powerful and — viewed sufficiently closely — ragged with illusions. Zoom in on individual experience, and something obscure from afar becomes obvious: in making our attentiveness a fungible asset, we’re not so much conjuring currency out of thin air as chronically undervaluing our time.

We watch a 30-second ad in exchange for a video; we solicit a friend’s endorsement; we freely pour sentence after sentence, hour after hour, into status updates and stock responses. None of this depletes our bank balances. Yet its cumulative cost, while hard to quantify, affects many of those things we hope to put at the heart of a happy life: rich relationships, rewarding leisure, meaningful work, peace of mind.

What kind of attention do we deserve from those around us, or owe to them in return? What kind of attention do we ourselves deserve, or need, if we are to be ‘us’ in the fullest possible sense? These aren’t questions that even the most finely tuned popularity contest can resolve. Yet, if contentment and a sense of control are partial measures of success, many of us are selling ourselves far too cheap.

Are you still paying attention? I can look for signs, but in the end I can’t control what you think or do. And this must be the beginning of any sensible discussion. No matter who or what tells you otherwise, you have the perfect right to ignore me — and to decide for yourself what waits in each waking moment.

The economy is more a messy, fractal living thing than a machine

 Mainstream economics is built on the premise that the economy is a machine-like system operating at equilibrium. According to this idea, individual actors – such as companies, government departments and consumers – behave in a rational way. The system might experience shocks, but the result of all these minute decisions is that the economy eventually works its way back to a stable state.

Unfortunately, this naive approach prevents us from coming to terms with the profound consequences of machine learning, robotics and artificial intelligence. Most economists’ predictions in the area are wildly unrealistic. On the one hand, liberals worry about increased income inequality and exclusion due to the automation of labour, and the pooling of poorly paid work on platforms such as Uber and TaskRabbit. They fret about the fact that only the highly intelligent, educated or creative will thrive. Governments keen to soften the blow are considering so-called helicopter money in the guise of universal basic income. But such strategies ignore humans’ fundamental need to feel needed, to be creative and productive, and to achieve status and acceptance in the eyes of their community. Any attempt to ‘buy’ citizen loyalty through a massive, rebranded welfare system risks creating further disaffection and instability.

Meanwhile, pundits further to the political Left indulge in fantasies of automated luxury communism, in which artificial intelligence, directed by a socialist-style government, makes work discretionary. But this scenario fails to explain how innovation will be sustained, or how the very costly information infrastructure will be maintained, when the only motivation to do so will be altruistic.

Both political camps accept a version of the elegant premise of economic equilibrium, which inclines them to a deterministic, linear way of thinking. But why not look at the economy in terms of the messy complexity of natural systems, such as the fractal growth of living organisms or the frantic jive of atoms? These frameworks are bigger than the sum of their parts, in that you can’t predict the behaviour of the whole by studying the step-by-step movement of each individual bit. The underlying rules might be simple, but what emerges is inherently dynamic, chaotic and somehow self-organising. Complexity economics takes its cue from these systems, and creates computational models of artificial worlds in which the actors display a more symbiotic and changeable relationship to their environments. Seen in this light, the economy becomes a pattern of continuous motion, emerging from numerous interactions. The shape of the pattern influences the behaviour of the agents within it, which in turn influences the shape of the pattern, and so on.

There’s a stark contrast between the classical notion of equilibrium and the complex-systems perspective. The former assumes rational agents with near-perfect knowledge, while the latter recognises that agents are limited in various ways, and that their behaviour is contingent on the outcomes of their previous actions. Most significantly, complexity economics recognises that the system itself constantly changes and evolves – including when new technologies upend the rules of the game.

Ever since the invention of the assembly line, corporations have been like medieval cities: building walls around themselves and then trading with other ‘cities’ and consumers. Companies exist because of the need to protect production from volatile market fluctuations, and because it’s generally more efficient to consolidate the costs of getting goods and services to market by putting them together under one roof. So said the British economist Ronald Coase in his paper ‘The Nature of the Firm’ (1937).

But now, in an era of Ubers-for-everything, companies are changing into platforms that enable, rather than enact, core business processes. The cost of reaching customers has dropped dramatically thanks to the ubiquity of digital networks, and production is being pushed outside the company wall, on to freelancers and self-employed contractors. Market and price fluctuations have been defanged as machine learning and predictive analytics help companies manage such ructions, and on-demand services for labour, office space and infrastructure allow them to be more responsive to changing conditions. Coase’s theory is nearing its expiry date.

The so-called ‘gig economy’ is only the beginning of a profound economic, social and political transformation. For the moment, these new ways of working are still controlled by old-style businesses models – platforms that essentially sell ‘trust’ via reviews and verification, or by plugging into existing financial and legal systems. Airbnb, eBay and Uber succeed in making money out of other people’s work and assets because they provide guarantees for good seller-buyer behaviour, while connecting to the ‘old world’ of banks, courts and government. But this hybrid model of doing digital business is about to change.

Blockchain technologies promise to replace these trusted third parties with a huge digital record book, spreading out organically across a network of computers that grows and changes but can’t be meddled with. It’s true that there are huge regulatory challenges, and it’s still not clear if and how the computational and algorithmic infrastructure of the blockchain can be maintained at scale. Nonetheless, such distributed systems might truly democratise work in a way that’s hard for us to imagine. By getting rid of middlemen, they’re likely to radically reduce transaction costs, and accelerate the mixing of many different actors in the new economy who have been freed from the grip of leaders or institutions.

Here’s an alternative vision of what the future of work might bring. Imagine, for example, a network of individuals and families interconnected via an intelligent electrical grid, which produces and supplies only as much energy as required, drawing on solar panels installed on the roofs of their homes. They use 3D printing and robotics to manufacture or grow most of what they need, and exchange knowledge and expertise with other networks of people, whose productivity has been boosted by smart machines and data analytics. Infrastructure is shared and maintained by economic exchanges over these same networks. Blockchain technologies could verify contracts, and enable a barter economy to flourish. By changing the rules of the economic game, the whole ‘economy’ could transform – not in the deterministic, business-as-usual way that neoclassical economics predicts, but in the creative, chaotic and dynamic way of complex systems.

Such bottom-up, dispersed models of economic organisation would ultimately challenge our current political institutions. Instead of thinking about the future as the expansion of universal welfare, we might look at it as a rejuvenation of politics, where power shifts away from concentrated economic interests and into the hands of the empowered multitudes. In this scenario, it’s possible that the automation of work will mean the demise of big corporations and the rise of digital, small-scale, distributed cottage industries. Over time, these collaborative networks might evolve into virtual city-states, and even replace physical nations as the units of political organisation and citizen loyalty. Contrary to what current events suggest, the future of our planet could be bright: home to a self-organising ecosystem of democratic communities, cooperating to boost happiness, prosperity, longevity and creativity.

Game boys

 From a vast subculture of gaming addicts in China, only a few go professional and get rich. Is the social cost worth it?

Five young men sit in the living room of a large, grubby apartment in downtown Shanghai, playing a computer game. The room is mostly quiet, except for the hum of the computers and the steady click and keyboard-tap of the players, who wear headphones that envelop their ears and slouch in worn office chairs. A rotund, bespectacled gamer known as PDD is the most outspoken of the bunch, periodically shouting Wo cao! (‘Fuck me!’) at a setback, or leaping out of his chair to smother the adjacent player’s face with his large blue shirt in a moment of triumph.

Aged in their teens and early 20s, they are professional players of League of Legends(LoL), a competitive online game in which two teams of mythical characters called ‘champions’ battle it out in a fantasy arena. The objective of the game is to destroy the enemy’s base through a series of intricate moves involving teamwork, strategy and the nimble use of keyboard and mouse. The stakes are real because victory could earn the players huge cash prizes.

The online warriors of Shanghai’s famous Invictus Gaming club, or iG, who live and train together in the slovenly apartment, are hardly typical of urban Chinese their age. But their stories reflect much of what’s exciting and surreal about the brave new world being created by China’s ballistic economic and technological ascent. From varied places and economic backgrounds, they have gathered in Shanghai to earn a living through ‘eSports’, or digital combat. Like millions of other Chinese, the gamers and their organisers are chasing success in ways their parents’ generation can barely comprehend.

In a live match in the Polish city of Katowice last March, these same five cyber athletes took on a London-based team called Fnatic. The four-day tournament, sponsored by Intel and other computer hardware companies, offered a $60,000 prize to the winning LoL team (other teams also competed at the game StarCraft II for a $100,000 prize). Wearing headphones and uniformed tracksuits or T‑shirts, each team of sober-faced gamers clicked and tapped away at a row of computers in a packed sports arena. The audience watched the gameplay on giant screens.

Four minutes into the match, PDD’s champion, Shyvana (‘the Half-Dragon’), defended a tower by lobbing crystal-like projectiles at critters called minions. Elsewhere on the map, Fnatic’s tower emitted purple fireballs while another iG champion, Thresh, swung his chain at a winged demon called Nocturne, delivering a lethal blow. Nocturne’s lifeline dropped to zero and he burst into blue flame as the words ‘FIRST BLOOD!’ appeared on the screen. After 20 seconds more of chaotic action, a dance of twisting avatars and flying missiles far too furious for the untrained eye to track, two of iG’s champions converged on a fleeing enemy, Lulu. One of the gamers, Kid, controlling a champion called Vayne, shot a toxic arrow at Lulu, killing her in a bloody and prolonged explosion. ‘Kid picks up the kill!’ shouted one of the European men who were giving the play-by-play.

The young men of iG – founded in 2011 by Wang Sicong, son of Wang Jianlin, a property and film tycoon and one of the wealthiest men in China – play for love of the game but also for money. The club is partly funded by Wang Sicong, but it also enjoys corporate sponsorship from Logitech, a computer accessories maker, and ASUS, a PC vendor. The iG club pays the gamers a base salary of about 4,000-5,000 yuan per month (around US$650-$800) and covers their room and board. On top of that, the gamers can rake in lavish prizes from live matches in China or abroad. In 2012, the division of the club that now plays Defense of the Ancients 2 (Dota 2), another wildly popular online game, split a $1 million first prize in Seattle. The prospect of even bigger winnings beckons; a Chinese team called Newbee took home $5 million from the same event last July.

The iG club’s manager Zhu Songge, who goes by the name Lucien, told me that one of the team members, Kid, earned roughly 400,000 yuan (about $64,000) last year – a princely sum for a 16-year-old from Henan province, where the average urban disposable income is about 5 per cent of that. The gamer, known as Ge Yan in real life, used the money to buy his family a house. Kid’s laconic, awkward manner belies the aggressiveness of his role in the game, which is to lead the team to victory by mowing down the enemy with deadly implements from crossbows to phosphorous bombs.

‘Sometimes I wouldn’t return home for dozens of days. Sometimes I would have the idea of hitting my parents’

Only the most outrageously skilled gamers – a tiny minority – can make a living this way. Lucien estimates that there are around 50 pro clubs for LoL in China, and 10 to 15 in Shanghai; each club has about five players. The elite, successful world of professional eSports draws its audience and its star players from a vast and problematic gaming subculture. For every well-paid Kid, there are tens of thousands of young gaming enthusiasts who play for fun. Many of them are compulsive gamers, logging countless hours in virtual worlds of adventure, fantasy and violence. Their habits are viewed by parents, teachers, and authorities mostly with disapproval and concern.

Far from Shanghai, in the city of Wuhan in central China, 17-year-old Zehao described his struggles with obsessive game-playing (in his case, involving the popular first-person shooter CrossFire) by telephone. ‘I used to be so addicted to the internet that I couldn’t free myself,’ he said. ‘Sometimes I wouldn’t even return home [from an internet café] for dozens of days. Sometimes I would have the idea of hitting my parents, or scolding them. I feel guilty about this now.’

The youth is a student of Tao Hongkai, a professor at Central China Normal University in Wuhan who counsels compulsive internet users. His mission is roughly the reverse of Lucien’s: to steer his charges away from online games. ‘I’m having class with them now,’ Tao said before handing the phone to Wang. ‘There are dozens of kids here with an internet addiction problem. One of them was always skipping class and not coming home because of his gaming. So his parents brought him here. After attending my class, he realised he would have to change his ways, because if he keeps playing like this he will destroy himself.’

China’s gaming scene is vast. The country’s internet population, which grows by tens of millions of people every year, includes an estimated 147 million hardcore online game users who spent more than $13 billion in 2013. In the process, they enrich companies such as the Shenzhen-based Tencent, owner of LoL’s US publisher and the largest gaming company in the world.

The heroes of the scene have good salaries and minor celebrity status, rewards that pushed them to go pro. One such gamer was Liu Hongjun, or ‘Kitties’. From a town near Chengdu, the capital of Sichuan province, the 21-year-old Kitties has an easy-going smile and a languid, friendly way of talking. A college graduate who studied pre-med with plans to become a doctor, he changed course when he discovered how much he loved gaming. At the time, he was earning 500 yuan (about $80) a month as an intern at a medical company, handling purchasing. He began playing LoL in his free time and quickly achieved a high rank, which brought him to the attention of a gaming team in Chengdu. The team recruited him and paid him 2,000 yuan in the first month. He later did a stint at another team before joining iG.

His feats of masterful tower-building and monster-slaying made him a minor national celebrity; he carried the flame during China’s 2008 Olympic torch relay

Kitties kept his new occupation secret from his parents until he started earning enough money to support himself. His father, who rests at home for health reasons, and his mother, who works at a factory, are pleased with his current career. Other people in his hometown are equally supportive – the money speaks for itself – but it’s not always easy for Kitties to make them understand what he does for a living. How does one explain multiplayer online battle arenas to an elderly relative in a small town in central China?

Li Xiaofeng, or ‘Sky’, a famous ex-professional gamer, faced a tougher path to success. Sky’s grades suffered from his tendency to slip away to internet cafés to play games as a middle-school student in the late 1990s. As a punishment, his father locked him in his room and beat him with a belt.

He eventually learned to put his talents to profitable use in the eSports scene, becoming the leading Chinese player of the strategy game Warcraft III. According to the website esportsearnings.com, Sky won more than $232,000 in tournament prize money from 2005 to 2012. His feats of masterful tower-building and monster-slaying made him a minor national celebrity; he carried the flame during China’s 2008 Olympic torch relay.

‘There isn’t this sense of, it’s okay for my kid to play eSports for 20 hours a day because he or she’s going to become a professional,’ said Marcella Szablewicz, assistant communications professor at Pace University in New York City and an expert on China’s internet gaming scene, when we met in Shanghai. ‘They talk about how [Sky] was one of those bad kids who spent all his time at an internet café, but then he became an eSports hero, and now he’s fantastic, and now we all love him. So how you get from that position to eSports athlete – there isn’t really a clear path.’

In a country where social mobility is closely linked to academic success, it’s understandable that most parents would rather see their kids hitting the books than honing their game-playing skills. A high score on the gaokao, China’s brutally competitive national college entrance exam, seems more attainable than a rarefied career as a salaried eSports athlete. But the pursuit of either goal requires total commitment.

When professional Chinese gamers are asked how to get involved in eSports, Szablewicz observes, they usually recommend finishing school first. ‘This is like the party line,’ she said, ‘because of course nobody finishes school first and then starts playing. The kids who get to be really good are not doing well in school probably.’

Ironically, the very intensity of the education system and the pressures it places on young Chinese people have helped to fuel the popularity of online gaming. ‘Really, this emanates from the internet café culture, which does not, to a large extent, exist in the United States,’ Szablewicz explained, referring to the eSports culture in both South Korea and China (eSports in the US evolved out of console gaming, unlike in China, where video game consoles were banned until 2014). ‘And I think that also the popularity of these internet cafés happened because of the fact that the school system here in China, and in Korea for that matter, is fairly restrictive. There isn’t much by way of cultivation of extracurricular activities. There’s so much focus on the college entrance exam.’

The phrase ‘electronic heroin’ has particular resonance in a country still touchy about mass opium addiction in the 19th century

Of course, concerns about excessive gameplay run deeper than parental angst over their kids’ grades. Authorities fear the social and psychological effects of mass online gaming. China was one of the first countries to label ‘internet addiction’ as a mental disorder, and hundreds of rehab centres, official and unofficial, have been set up to treat compulsive internet use, including harsh boot camps where adolescents are deprived of computers and subjected to military-style drills.

In 2007, Beijing issued new rules requiring gamers and other internet users to register with their real names and ID numbers. China forced game operators to install a ‘fatigue system’, in which players under 18 saw their points cut in half after three hours of play, and reduced to zero after five hours. A temporary nationwide ban on the opening of new internet cafés was imposed at around the same time.

The phrase ‘electronic heroin’, often used to describe online gaming, has particular resonance in a country still touchy over the disasters that mass opium addiction wrought in the 19th century. The fear is bolstered by disturbing lines of research showing that obsessive online gaming can cause neurological changes similar to those observed in drug addicts, including damage to the dopamine reward system.

State media report that more than 24 million young people in China are addicted to the internet, and officials have seized those numbers to explain many of the nation’s woes: compulsive web use is widely claimed to be a leading cause of mental illness, moral decay, and even juvenile delinquency. In 2005, a prominent Beijing judge reckoned that 90 per cent of youth crime in the city was related to internet addiction, and Tao, the teacher in Wuhan, cited similar figures. Whether or not such alarming claims have merit, binge-gaming is clearly a problem for the many people whose lives have been disrupted or destroyed by it. Reports of people dying after continuous, days-long gaming sessions hint at the possible dangers, as does a Chinese media report last July that a young, unmarried couple sold their two infant sons to child traffickers to help fund their purchases of in-game virtual items.

‘Scientists have found that playing computer games over the long term… can cause loss of emotional control’ in adolescents, Tao told me. ‘That’s why some people would rather play games than eat or go home. More than 20 students here have this problem – they don’t go home, they argue with their parents, and some even hit their parents.’ In another conversation, Tao said that internet addiction ‘is the main problem for all the families and schools’.

Experts such as Szablewicz, of course, find it difficult to blame online gaming for quite so much. She questions the alleged link between hardcore gaming and youth crime: ‘That’s the kind of really problematic language that happens here in the Chinese press, I think. The thing is that so many young people today are internet gamers or they play games, that of course if they track every single kid who’s ever played a game, they’re going to find these connections. It doesn’t necessarily suggest causality.

‘I don’t deny that there are kids with problematic behaviours with regard to gaming,’ she added. ‘I’m very critical of the term internet addiction, but I’ve talked to kids who’ve played 72 hours straight without standing up, you know, things like that. Maybe they use the bathroom, that’s it. It can be problematic.’

Observing iG’s members training in the apartment, I wondered what separated them from the Chinese adolescents hauled off to internet addiction boot camps. The gamers sharpen their skills at the computer for 12 to 14 hours each day. Their focus on the game is so relentless that I found most of them riveted to their screens even on a Sunday, their day off – sometimes goofing around or watching funny videos on the internet, but mostly just playing LoL. A headphone-wearing, chain-smoking PDD basically ignored his girlfriend, who lives in another room in the building, whenever she walked in to grab some computer equipment or bring him a snack. ‘I’ll play games my whole life,’ PDD told me at one point. ‘I think I can never stop.’

But playing at that level does, after all, take extreme physical and mental dedication; and I inevitably missed much during my few visits. ‘We’re young, we like sports,’ Lucien told me over dinner with his girlfriend Vivian, an eSports commentator, and Szablewicz one Saturday. ‘We like basketball, football. We like KTV [karaoke]. We don’t stare at our computer every day and play, play, play.’ Lucien pointed out that the team was going swimming the next day. ‘You can think of us as a football club – it’s all the same.’

The rise of powerhouse teams such as iG explains why the same government that wages war on gaming addiction has also recognised eSports as an official sport since 2003. State organisations help to run or sponsor gaming tournaments such as the annual World Cyber Games event, held in the Chinese city of Kunshan in 2012 and 2013. China’s State General Administration of Sport even has a bureau that oversees the development of eSports culture.

This divided approach to internet gaming has its own strange logic. So-called multiplayer online battle arena (MOBA) games such as LoL and Dota 2 – two of the most popular eSports titles in China – and real-time strategy (RTS) games such as Starcraft, which have start and stop points, are viewed as distinct from and more legitimate than massively multiplayer online role-playing games (MMORPGs) such as World of Warcraft, which continue endlessly. ‘Generally, the games that are seen as being most harmful and most addictive are these endless, massively multiplayer online role-playing games,’ Szablewicz said. ‘They’re seen as aimless, like they’re allowing young people to give into the fantasy of these virtual worlds. An eSports game generally lasts between 20 and 40 minutes.’ Of course, she added, that doesn’t stop anyone from playing one game after another, chain-smoking style.

More importantly, the Chinese government has learned not to block the rise of a thriving industry. A club such as iG is in some ways a prototype of China’s new economy – with an atmosphere more like a tech start-up than a home for troubled junkies. The slim and youthful Lucien is the polished, professional face of the club, handling iG’s business and relationships with sponsors, and supervising the players and coaches. A graduate of Shanghai’s prestigious Fudan University Law School, he started a law firm before switching careers. Lucien talks in terms that would be familiar to any young entrepreneur or ambitious professional in the West (or in China, for that matter). He wants to help develop the emerging eSports industry domestically, which he sees as more promising than law owing to its novelty and rapid growth. Lucien hopes to start an eSports school where gamers can train and study academic subjects all at once.

after age 22 or 23, gamers’ reflexes and eye-hand co‑ordination decline so much they must retire

The club’s LoL coach, an affable 26-year-old who goes by the name Snow, also comes across as a white-collar striver. Snow sits at a single desk in the back of the training room, wearing large white headphones with ear cups that pulse blue light. While the team members train, he watches and records their gameplay on his computer; afterward, they gather around his desk to review the videos and get feedback.

Snow, who wears sweatshirts and has an unruly mass of hair, is soft-spoken but reflects on his gaming life in rapid, articulate sentences that suggest he’s given it a lot of thought. After studying industrial engineering at college, Snow ventured into finance; he ran a store on Taobao, China’s giant e-commerce platform, and worked in currency exchange, but found he wasn’t really cut out for these jobs.

In 2013, another career option presented itself to Snow, who was a fan of iG: though too old by then to compete professionally (after age 22 or 23, gamers’ reflexes and eye-hand co‑ordination decline so much they must retire), he decided to try his hand at coaching. Along with related jobs such as managing and sports-casting, coaching is a common path for former gamers past their use-by date, as well as an option for enthusiasts such as Snow. ‘Basically, I tried different ways to improve this team, but finally found that these players are very talented and have their own thoughts,’ Snow told me. ‘I think I will hurt their understanding of the game if I try to impose my ideas on them by telling them what to do… I don’t interfere much during the training, and I let them bring their talent into full play.’

The rise of professional gaming in China creates new opportunities for highly gifted players, as well as for people such as Lucien and Snow, who bring deep insight and service-sector skills to a fast-rising new industry. For them, eSports offers yet another vehicle for ambitious dreams in an ever-expanding galaxy of choices. In the new world of China, every dreamer has a shot – and even the lowliest gaming addict might one day be a star.

Slaves to the algorithm

 Computers could take some tough choices out of our hands, if we let them. Is there still a place for human judgment?

In central London this spring, eight of the world’s greatest minds performed on a dimly lit stage in a wood-panelled theatre. An audience of hundreds watched in hushed reverence. This was the closing stretch of the 14-round Candidates’ Tournament, to decide who would take on the current chess world champion, Viswanathan Anand, later this year.

Each round took a day: one game could last seven or eight hours. Sometimes both players would be hunched over their board together, elbows on table, splayed fingers propping up heads as though to support their craniums against tremendous internal pressure. At times, one player would lean forward while his rival slumped back in an executive leather chair like a bored office worker, staring into space. Then the opponent would make his move, stop his clock, and stand up, wandering around to cast an expert glance over the positions in the other games before stalking upstage to pour himself more coffee. On a raised dais, inscrutable, sat the white-haired arbiter, the tournament’s presiding official. Behind him was a giant screen showing the four current chess positions. So proceeded the fantastically complex slow-motion violence of the games, and the silently intense emotional theatre of their players.

When Garry Kasparov lost his second match against the IBM supercomputer Deep Blue in 1997, people predicted that computers would eventually destroy chess, both as a contest and as a spectator sport. Chess might be very complicated but it is still mathematically finite. Computers that are fed the right rules can, in principle, calculate ideal chess variations perfectly, whereas humans make mistakes. Today, anyone with a laptop can run commercial chess software that will reliably defeat all but a few hundred humans on the planet. Isn’t the spectacle of puny humans playing error-strewn chess games just a nostalgic throwback?

Such a dismissive attitude would be in tune with the spirit of the times. Our age elevates the precision-tooled power of the algorithm over flawed human judgment. From web search to marketing and stock-trading, and even education and policing, the power of computers that crunch data according to complex sets of if-then rules is promised to make our lives better in every way. Automated retailers will tell you which book you want to read next; dating websites will compute your perfect life-partner; self-driving cars will reduce accidents; crime will be predicted and prevented algorithmically. If only we minimise the input of messy human minds, we can all have better decisions made for us. So runs the hard sell of our current algorithm fetish.

If we let cars do the driving, we are outsourcing not only our motor control but also our moral judgment

But in chess, at least, the algorithm has not displaced human judgment. The imperfectly human players who contested the last round of the Candidates’ Tournament — in a thrilling finish that, thanks to unusual tiebreak rules, confirmed the 22-year-old Norwegian Magnus Carlsen as the winner, ahead of former world champion Vladimir Kramnik — were watched by an online audience of 100,000 people. In fact, the host of the streamed coverage, the chatty and personable international master Lawrence Trent, pointedly refused to use a computer engine (which he called ‘the beast’) for his own analyses and predictions. The idea, he explained, is to try to figure things out for yourself. During a break in the commentary room on the day I was there, Trent was eating crisps and still eagerly discussing variations with his plummily amusing co-presenter, Nigel Short (who himself had contested the World Championship against Kasparov in 1993). ‘He’ll find Qf4; it’s not difficult to find,’ Short assured Trent. ‘Ng8, then it’s…’ ‘It’s game over.’ ‘Game over!’

Chess is an Olympian battle of wits. As with any sport, the interest lies in watching profoundly talented humans operating at the limits of their capability. There does exist a cyborg version of the game, dubbed ‘advanced chess’, in which humans are allowed to use computers while playing. But it is profoundly boring to watch, like a contest over who can use spreadsheet software more effectively, and hasn’t caught on. The ‘beast’ can be a useful helpmeet — Veselin Topalov, a previous challenger for Anand’s world title, used a 10,000-CPU monster in his preparation for that match, which he still lost — but it’s never going to be the main event.

This is a lesson that the algorithm-boosters in the wider culture have yet to learn. And outside the Platonically pure cosmos of chess, when we seek to hand over our decision-making to automatic routines in areas that have concrete social and political consequences, the results might be troubling indeed.

At first thought, it seems like a pure futuristic boon — the idea of a car that drives itself, currently under development by Google. Already legal in Nevada, Florida and California, computerised cars will be able to drive faster and closer together, reducing congestion while also being safer. They’ll drop you at your office then go and park themselves. What’s not to like? Well, for a start, as the mordant critic of computer-aided ‘solutionism’ Evgeny Morozov points out, the consequences for urban planning might be undesirable to some. ‘Would self-driving cars result in inferior public transportation as more people took up driving?’ he wonders in his new book, To Save Everything, Click Here (2013).

More recently, Gary Marcus, professor of psychology at New York University, offered a vivid thought experiment in The New Yorker. Suppose you are in a self-driving car going across a narrow bridge, and a school bus full of children hurtles out of control towards you. There is no room for the vehicles to pass each other. Should the self-driving car take the decision to drive off the bridge and kill you in order to save the children?

What Marcus’s example demonstrates is the fact that driving a car is not simply a technical operation, of the sort that machines can do more efficiently. It is also a moral operation. (His example is effectively a kind of ‘trolley problem’, of the sort that has lately been fashionable in moral philosophy.) If we let cars do the driving, we are outsourcing not only our motor control but also our moral judgment.

Meanwhile, as Morozov relates, a single Californian company called Impermium provides software to tens of thousands of websites to automatically flag online comments for ‘not only spam and malicious links, but all kinds of harmful content — such as violence, racism, flagrant profanity, and hate speech’. How do Impermium’s algorithms decide exactly what should count as ‘hate speech’ or obscenity? No one knows, because the company, quite understandably, isn’t going to give away its secrets. Yet rather than pursuing mere lexicographical analysis, such a system of automated pre-censorship is, again, making moral judgments.

If self-driving cars and speech-policing systems are going to make hard moral decisions for us, we have a serious stake in knowing exactly how they are programmed to do it. We are unlikely to be content simply to trust Google, or any other company, not to code any evil into its algorithms. For this reason, Morozov and other thinkers say that we need to create a class of ‘algorithmic auditors’ — trusted representatives of the public who can peer into the code to see what kinds of implicit political and ethical judgments are buried there, and report their findings back to us. This is a good idea, though it poses practical problems about how companies can retain the commercial edge provided by their computerised secret sauce if they have to open up their algorithms to quasi-official scrutiny.

If we answer yes, we are giving our blessing to something even more nebulous than thoughtcrime. Call it ‘unconscious brain-state crime’

A further problem is that some algorithms positively must be kept under wraps in order to work properly. It is already possible, for example, for malicious operators to ‘game’ Google’s autocomplete results — sending abusive or libellous descriptions to the top of Google’s suggestions when you type a person’s name — and lawsuits from people affected in this way have already forced the company to delve into the system and change such examples manually. If it were made public exactly how Google’s PageRank algorithm computes the authority of web pages, or how Twitter’s ‘trending’ algorithm determines the popularity of subjects, then unscrupulous self-marketers or vengeful exes would soon be gaming those algorithms for their own purposes too. The vast majority of users would lose out, because the systems would become less reliable.

And it doesn’t necessarily require a malicious individual gaming a system for algorithms to get uncomfortably personal. Automatic analysis of our smartphone geolocation, internet-browsing and social-media data-trails grows ever more sophisticated, and so we can thin-slice demographic categories ever more precisely. From such information it is possible to infer personal details (such as sexual orientation or use of illegal drugs) that have not been explicitly supplied, and sometimes to identify unique individuals. Even when such information is simply used to target adverts more accurately, the consequences can be uncomfortable. Last year, the journalist Charles Duhigg related a telling anecdote in an article for The New York Times called ‘How Companies Learn Your Secrets’. A decade ago, the American retailer Target sent promotional baby-care vouchers to a teenage girl in Minneapolis. Her father was so outraged, he went to the shop to complain. The manager was equally taken aback and apologised; a few days later, he called the family to apologise again. This time, it was the father who offered an apology: his daughter really was pregnant, and Target’s ‘predictive analytics’ system knew it before he did.

Such automated augury might be considered relatively harmless if its use is confined to figuring out what products we might like to buy. But it is not going to stop there. One day in the near future — perhaps this has already happened — an innocent crime novelist researching bloody techniques for his latest fictional serial killer will find armed men banging on his door in the middle of the night, because he left a data trail that caused lights to flash red in some preventive-policing algorithm. Perhaps a few distressed writers is a price we are willing to pay to prevent more murders. But predictive crime prevention is an area that leads rapidly to a dystopian sci-fi vision like that of the film Minority Report (2002).

In Baltimore and Philadelphia, software is already being used to predict which prisoners will reoffend if released. The software works on a crime database, and variables including geographic location, type of crime previously committed, and age of prisoner at previous offence. In so doing, according to a report in Wired in January this year, ‘The software aims to replace the judgments parole officers already make based on a parolee’s criminal record.’ Outsourcing this kind of moral judgment, where a person’s liberty is at stake, understandably makes some people uncomfortable. First, we don’t yet know whether the system is more accurate than humans. Secondly, even if it is more accurate but less than completely accurate, it will inevitably produce false positives — resulting in the continuing incarceration of people who wouldn’t have reoffended. Such false positives undoubtedly occur, too, in the present system of human judgment, but at least we might feel that we can hold those making the decisions responsible. How do you hold an algorithm responsible?

Still more science-fictional are recent reports claiming that brain scans might be able to predict recidivism by themselves. According to a press release for the research, conducted by the American non-profit organisation the Mind Research Network, ‘inmates with relatively low anterior cingulate activity were twice as likely to reoffend than inmates with high-brain activity in this region’. Twice as likely, of course, is not certain. But imagine, for the sake of argument, that eventually a 100 per cent correlation could be determined between certain brain states and future recidivism. Would it then be acceptable to deny people their freedom on such an algorithmic basis? If we answer yes, we are giving our blessing to something even more nebulous than thoughtcrime. Call it ‘unconscious brain-state crime’. In a different context, such algorithm-driven diagnosis could be used positively: according to one recent study at Duke University in North Carolina, there might be a neural signature for psychopathy, which the researchers at the laboratory of neurogenetics suggest could be used to devise better treatments. But to rely on such an algorithm for predicting recidivism is to accept that people should be locked up simply on the basis of facts about their physiology.

If we erect algorithms as our ultimate judges and arbiters, we face the threat of difficulties not only in law-enforcement but also in culture. In the latter realm, the potential unintended consequences are not as serious as depriving an innocent person of liberty, but they still might be regrettable. For if they become very popular, algorithmic systems could end up destroying what they feed on.

In the early days of Amazon, the company employed a panel of book critics, whose job was to recommend books to customers. When Amazon developed its algorithmic recommendation engine — an automated system based on data about what others had bought — sales shot up. So Amazon sacked the humans. Not many people are likely to weep hot tears over a few unemployed literary critics, but there still seems room to ask whether there is a difference between recommendations that lead to more sales, and recommendations that are better according to some other criterion — expanding readers’ horizons, for example, by introducing them to things they would never otherwise have tried. It goes without saying that, from Amazon’s point of view, ‘better’ is defined as ‘drives more sales’, but we might not all agree.

Algorithmic recommendation engines now exist not only for books, films and music but also for articles on the internet. There is so much out there that even the most popular human ‘curators’ cannot possibly keep on top of all of it. So what’s wrong with letting the bots have a go? Viktor Mayer-Schönberger is professor of internet governance and regulation at Oxford University; Kenneth Cukier is the data editor of The Economist. In their book Big Data (2013) — which also calls for algorithmic auditors — they sing the praises of one Californian company, Prismatic, that, in their description, ‘aggregates and ranks content from across the Web on the basis of text analysis, user preferences, social-network-related popularity, and big-data analytics’. In this way, the authors claim, the company is able to ‘tell the world what it ought to pay attention to better than the editors of The New York Times’. We might happily agree — so long as we concur with the implied judgment that what is most popular on the internet at any given time is what is most worth reading. Aficionados of listicles, spats between technology theorists, and cat-based modes of pageview trolling do not perhaps constitute the entire global reading audience.

So-called ‘aggregators’ — websites, such as the Huffington Post, that reproduce portions of articles from other media organisations — also deploy algorithms alongside human judgment to determine what to push under the reader’s nose. ‘The data,’ Mayer-Schönberger and Cukier explain admiringly, ‘can reveal what people want to read about better than the instincts of seasoned journalists’. This is true, of course, only if you believe that the job of a journalist is just to give the public what it already thinks it wants to read. Some, such as Cass Sunstein, the political theorist and Harvard professor of law, have long worried about the online ‘echo chamber’ phenomenon, in which people read only that which reinforces their currently held views. Improved algorithms seem destined to amplify such effects.

Some aggregator sites have also been criticised for paraphrasing too much of the original article and obscuring source links, making it difficult for most readers to read the whole thing at the original site. Still more remote from the source is news packaged by companies such as Summly — the iPhone app created by the British teenager Nick D’Aloisio — which used another company’s licensed algorithms to summarise news stories for reading on mobile phones. Yahoo recently bought Summly for $USD30 million. However, the companies that produce news often depend on pageviews to sell the advertising that funds the production of their ‘content’ in the first place. So, to use algorithm-aided aggregators or summarisers in daily life might help to render the very creation of content less likely in the future. In To Save Everything, Click Here, Evgeny Morozov draws a provocative analogy with energy use:

Our information habits are not very different from our energy habits: spend too much time getting all your information from various news aggregators and content farms who merely repackage expensive content produced by someone else, and you might be killing the news industry in a way not dissimilar from how leaving gadgets in the standby mode might be quietly and unnecessarily killing someone’s carbon offsets.

Meanwhile in education, ‘massive open online courses’ known as MOOCs promise (or threaten) to replace traditional university teaching with video ‘lectures’ online. The Silicon Valley hype surrounding these MOOCs has been stoked by the release of new software that automatically marks students’ essays. Computerised scoring of multiple-choice tests has been around for a long time, but can prose essays really be assessed algorithmically? Currently, more than 3,500 academics in the US have signed an online petition that says no, pointing out:

Computers cannot ‘read’. They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organisation, clarity, and veracity, among others.

It would not be surprising if these educators felt threatened by the claim that software can do an important part of their job. The overarching theme of all MOOC publicity is the prospect of teaching more people (students) using fewer people (professors). Will what is left really be ‘teaching’ worth the name?

One day, the makers of an algorithm-driven psychotherapy app could be sued by the survivors of someone to whom it gave the worst possible advice.

If you are feeling gloomy about the automation of higher education, the death of newspapers, and global warming, you might want to talk to someone — and there’s an algorithm for that, too. A new wave of smartphone apps with eccentric titular orthography (iStress, myinstantCOACH, MoodKit, BreakkUp) promise a psychotherapist in your pocket. Thus far they are not very intelligent, and require the user to do most of the work — though this second drawback could be said of many human counsellors too. Such apps hark back to one of the legendary milestones of ‘artificial intelligence’, the 1960s computer program called ELIZA. That system featured a mode in which it emulated Rogerian psychotherapy, responding to the user’s typed conversation with requests for amplification (‘Why do you say that?’) and picking up — with its ‘natural-language processing’ skills — on certain key words from the input. Rudimentary as it is, ELIZA can still seem spookily human. Its modern smartphone successors might be diverting, but this field presents an interesting challenge in the sense that, the more sophisticated it gets, the more potential for harm there will be. One day, the makers of an algorithm-driven psychotherapy app could be sued by the survivors of someone to whom it gave the worst possible advice.

What lies behind our current rush to automate everything we can imagine? Perhaps it is an idea that has leaked out into the general culture from cognitive science and psychology over the past half-century — that our brains are imperfect computers. If so, surely replacing them with actual computers can have nothing but benefits. Yet even in fields where the algorithm’s job is a relatively pure exercise in number- crunching, things can go alarmingly wrong.

Indeed, a backlash to algorithmic fetishism is already under way — at least in those areas where a dysfunctional algorithm’s effect is not some gradual and hard-to-measure social or cultural deterioration but an immediate difference to the bottom line of powerful financial organisations. High-frequency trading, where automated computer systems buy and sell shares very rapidly, can lead to the price of a security fluctuating wildly. Such systems were found to have contributed to the ‘flash crash’ of 2010, in which the Dow Jones index lost 9 per cent of its value in minutes. Last year, the New York Stock Exchange cancelled trades in six stocks whose prices had exhibited bizarre behaviour thanks to a rogue ‘algo’ — as the automated systems are known in the business — run by Knight Capital; as a result of this glitch, the company lost $440 million in 45 minutes. Regulatory authorities in Europe, Hong Kong and Australia are now proposing rules that would require such trading algorithms to be tested regularly; in India, an algo cannot even be deployed unless the National Stock Exchange is allowed to see it first and decides it is happy with how it works.

Here, then, are the first ‘algorithmic auditors’. Perhaps their example will prompt similar developments in other fields — culture, education, and crime — that are considerably more difficult to quantify, even when there is no immediate cash peril.

A casual kind of post-facto algorithmic auditing was already in evidence in London, at the Candidates’ Tournament. All the chess players gave press conferences after their games, analysing critical positions and showing what they were thinking. This often became a second contest in itself: players were reluctant to admit that they had missed anything (‘Of course, I saw that’), and vied to show they had calculated more deeply than their adversaries. On the day I attended, the amiable Anglophile Russian player (and cricket fanatic) Peter Svidler was discussing his colourful but peacefully concluded game with Israel’s Boris Gelfand, last year’s World Championship challenger. Juggling pieces on a laptop screen with a mouse, Svidler showed a complicated line that had been suggested by someone using a computer program. ‘This, apparently, is a draw,’ Svidler said, ‘but there’s absolutely no way anyone can work this out at the board’. The computer’s suggestion, in other words, was completely irrelevant to the game as a sporting exercise.

Now, as the rumpled Gelfand looked on with friendly interest, Svidler jumped to an earlier possible variation that he had considered pursuing during their game, ending up with a baffling position that might have led either to spectacular victory or chaotic defeat. ‘For me,’ he announced, ‘this will be either too funny … or not funny enough’. Everyone laughed. As yet, there is no algorithm for wry comedy.