Imagine this scene: In a vast concrete arena somewhere in Asia or South America, a silent game is being played out. It is probably, for the sake of argument, soccer, but it might as well be basketball or tennis or baseball or cricket or some as-yet-uninvented but already codified and anticipated combination of all of them.
In the stands, legions of robot fans look on. Articulated limbs raise LED placards in unison at appropriate moments, their faceplates shining with the avatars of distant, telepresent humans. On the field, two hologrammatic teams are composed entirely of flickering light: In silence, they sweep from one end to the other, reenacting a game which has already been pre-calculated and pre-played. Nothing is unexpected, no move unanticipated. Nobody is injured. The perfect game ends in a draw and the robots politely applaud the apotheosis of fully automated sporting prowess.
The technology to produce such an event is not merely already here, it is being implemented and advocated; and, while such a combination has not yet been explicitly proposed, it satisfies both the dictates of convergence and disruption beloved of technological determinism, and the political precepts of security and capitalization. Maximum security and minimum outlay combine to produce outrageous spectacle, even if it’s one which, from the outside, appears literally drained of blood.
Last month, the Hanwha Eagles, a struggling Korean baseball team, installed two dozen robot spectators in the middle of the home stand. The Eagles have lost 400 games in the last five years, and disconsolate supporters have started staying away. The effort to bring them back includes the ability for fair-weather fans to upload their faces to the team’s website and have it displayed on a robot’s visor, while the machine participates in fellow fans’ lackluster cheers and waves. The system plays into enough hi-tech east Asian stereotypes to be laughable for now–one English supporter quoted by the BBC joked that the robots might still revert to type and “misbehave “¦ get aggressive, abusive, spill a drink”¦ a robot hooligan!”–but is really in the vanguard of a trend for telepresence which covers a spectrum from Edward Snowden’s telepresent TED talk to the floating head of your micro-managing boss.
Meanwhile, Japan’s unsuccessful bid for the 2022 World Cup included the proposal to holographically broadcast matches to stadiums around the world, allowing for a “universal fan fest in 208 nations” as each game plays out simultaneously in every participating country, on actual soccer fields surrounded by cheering fans. The proposal also included the deeply frightening spectacle of a “revolutionary stadium experience,” where well-scrubbed fans invade the deserted field to snap selfies with hologrammatic samba dancers. It’s unclear whether the entirely unproven technology required to deliver this vision was the reason for the bid’s failure–although it was probably no less fanciful than the winner Qatar’s promises to lower the on-field air temperature by 36°F (20°C) and not kill hundreds of enslaved construction workers each year.
The combination of these two technologies–telepresent supporters and reconstructed players–sets the scene for our original scenario, but it doesn’t in the end go nearly far enough. The bloodlessness of the spectacle is present everywhere now, and is in correspondence to the increasing disparity between the actuality of what occurs and the “reality” we are presented with. How much of what you see on the field today is “real”? What counts as real, anyway?
In the situation presented above, we are watching a representation of a real event, an actual sporting event which is occurring elsewhere in real time. But even when we are, in person or remotely, watching a “real” sporting event, that event is ever more frequently a composite of practice and simulation; a mathematical calculation of a live event.
Another of the Japan 2022 promises was something called “Freeviewpoint Vision,” a system which would capture, down to the tiniest degree, every movement of the ball and the player on the field. In turn, the data gathered would permit both the hologrammatic fantasy game and an entirely new kind of sports cinematography–where the camera, loosened from its perch on the stands or strung on wires overhead, could swoop and duck around the field, passing between highly accurate recreations of the players, swinging beneath their legs, and following the ball right into the back of their legs. New, impossible images could be created, akin to the explosive imaginary of a Roland Emmerich film.
Freeviewpoint, unlike the holograms, is already on the market, and its origins are suitably Emmerich-like. Entrepreneur Aviv Shapira formerly worked for Israeli defense company Elbit Systems, where he led flight trials of military UAVs and developed the Watchkeeper drone for the British military. Appropriately, Shapira’s subsequent venture, Replay Technologies, launched its FreeD Video technology at the heavily surveilled London Olympics in 2012, with immersive recreations of the men’s gymnastics event.
In its promotional material, high-bouncing vaulters are frozen in mid-air, atomized into three-dimensional point clouds, and spun, rotated, and upended in time. Coaches and athletes look on in amazement. “It’s magic,” they say, over and over again. Replay Technologies has continued with Matrix-style, bullet-time renderings at the NBA All-Star’s Slam Dunk competition and Major League Baseball. In March, they were acquired for around $175 million by semiconductor giant Intel, underlining the computationally intensive nature of the technology. This is a spectacle produced in servers, not on sports fields.
What is “seen” is not “real”: It is a recreation, the appearance of something which may possibly have occurred but was not witnessed, only inferred. A Pepper’s Ghost for the Olympic age.
The uncanny nature of this representation is akin to the deep weirdness being produced by image processing in other visual applications. In October of 2014, AI researcher Robert Elliott Smith was reviewing an album of photos of himself and his wife on his Google+ account, when he chanced upon something disturbing: a photograph of “a moment which never happened.” Out of two similar photos of Smith and his wife on vacation in France, one in which she was smiling but he was not, and another where the situation was reversed, Google’s “AutoAwesome” algorithm had summoned a third: a composite image where both subjects were smiling at the same time. Smith titled the blog post exploring what had happened “It’s official: AIs are now rewriting history,” and the same could be applied to the sports field seen through the lens of FreeD.
This distinction between the actuality of the event and the fidelity of its recreation is narrow and could easily be dismissed as just another conjuration of spectacular TV coverage, were its remit limited to mere representation. But in the hyper-competitive domain of sports, lubricated with broadcasting and gambling dollars, recreation turns into prediction, and representation into judgement. The distinction between what is seen and what occurs becomes crucial.
More and more, the practice of human adjudication in sports is being crowded out by the supposed superiority of machine perception; a perception which is based on the recreation and prediction of real events, rather than their explicit witnessing. Since 2001, the Hawk-Eye computer system has become increasingly ubiquitous in major sporting competitions, combining machine vision with motion analysis to not only declare where precisely a ball touched or crossed a line, but where the ball would have gone if it were not rudely interrupted.
Hawk-Eye not only now makes the line calls in tennis and determines when a soccer ball has crossed the goal line, it also decides on future events. In cricket, a batsman can be dismissed if his leg, rather than his bat, deflects the ball away from the wicket. Hawk-Eye’s algorithms model the path of the ball and produce a trajectory showing whether or not it would have struck the wicket. Something which did not occur is being summoned into reality to determine the status quo. Like quantum physics’ famous two-slit apparatus–which proved that light behaves as both a wave and a particle, and in unpredictable, probabilistic ways–Hawk-Eye and systems like it are essentially thought experiments, but ones which determine the outcome of reality.
This kind of evocation is not limited to the field of play either. Unlike broadcast cameras, Hawk-Eye, FreeD, and similar systems are not mere witnesses to the game; they are recording devices, mapping and storing every moment of play for later recall and analysis. One of the sporting cultures which has most completely embraced the use of machine vision and analytics is U.S. basketball, whose governing body the NBA has arranged for every arena to be festooned with cameras. These cameras are part of the SportVU system, which tracks the x, y, and z coordinates of every player, as well as the ball, capturing a complete picture of the game 25 times a second. In a ghoulish but hardly surprising twist, SportVU was founded by another Elbit graduate, Israeli scientist Miky Tamir, whose previous expertise was in missile tracking. (I’ll stop talking about Elbit now, I promise, but if you want to see one more crossover, check out this footage of Neymar’s opening goal in the 2013 Confederations Cup, as filmed directly by an Elbit Hermes drone–aka the Watchkeeper.)
In the NBA, the integration of SportVU has led to an almost complete inversion of coaching strategies: Rather than training players to lead and direct play, they are trained to anticipate and respond to the predicted behaviors of their opponents. Of course, sportspeople have always checked out the opposition. Coaches watch videos of previous games and note common plays and strategies; goalkeepers obsessively review shots by opposing strikers, alaysing their body language and angle of attack. But SportVU amplifies and optimizes this process. Every characteristic of both teammate and opponent can be pored over and filed away. It can also be simulated.
In 2013, the Toronto Raptors revealed that they had built their own software on top of the league-wide SportVU data to simulate optimal player behaviors. By analyzing every single play, they worked out where their players had fouled, missed passes or shots, been short of the mark, or feinted right when they should have gone left. They looked at their opponents, too: saw where they hesitated, where they held back, where an advantage could have been pressed, or a tackle dodged. They unearthed the hidden, unplayed, but possible game within the game; an ideal game, but one which they won, obviously. In footage the Raptors’ analytics team showed from its system, each player is shadowed by what is referred to as a “ghost player,” one who appears to be always one step ahead, one step closer to a goal, one shot better than the real thing. The ghost player is a computationally augmented version of the real thing, one who makes better decisions and better plays. One who wins the game. The ghost player does what the player didn’t, but should have, done.
The concept of the “ghost car” is familiar from computer games: an ethereal representation of one’s best performance, almost always infuriatingly, unattainably ahead, except in the rare moment you break through, scoot past, and set a new high score, a new yardstick of attainment. It’s a race against oneself, an attempt to better oneself, to improve, to self-perfect. But the Raptors’ ghost players are something very different. They are players who have never existed, might never exist, but whose performance can be benchmarked and emulated. Instead of bettering themselves, the players can be trained to better resemble the meticulous creation of the algorithm: zombies playing out an imperfect recreation of an already-determined scenario.
The gap between what can be predicted and what can be acted on is closing fast. Algorithmic commands don’t need to be pre-programmed but can be determined once the match is already in progress. Sportstech LLC, a company founded by an astrophysics professor and his programmer son, has filed patents for in-game predictions: a motion-tracking system capable of alerting players to a shot’s accuracy at the very moment it is fired. Potential applications include a flashing light to tell basketball players a shot is on target before it lands, so that the defenders could immediately go offensive without bothering to actually play any more defense, and a vibrating wristband for goalkeepers which would prevent them from conceding corners by deflecting balls which weren’t going in anyway. Who needs team practice when you have machine-augmented precognition beamed right into your sensorium?
As image recognition, motion capture, and raw computational capacity improves, it’s obvious that the trajectory is toward not better endurance or ball skill, but better prediction and emulation. Players become actors–or reenactors–and the sports stars of the future will be those who can best and most quickly adapt themselves to the dictates of algorithmic play. Likewise, the successful teams–if they aren’t already, and some of them surely are–will be those with the most programmers on side, the most AI techs, the best simulation teams, and the best visualizers. Ultimately, perhaps, we will reach the perfect state described in our opening scenario: the ideal game which tends, inevitably and mathematically–and unfortunately for the spectators–toward the draw.
Perfect pass, perfect tackle, locked in a computationally impossible cycle. The only winning move is not to play.
A friend recounted a story about playing Facebook Scrabble. Bored, eventually, with losing, they started surreptitiously referring to a cheat site, which calculated the best possible move from the available tiles. Shamefully at first, and gradually out of habit, they referred every move to the web page, and started winning, not every time, but definitely more often. It was convenient and efficient, too: the yawning abyss of patient mental application was more profitably filled with more clicks, more pageviews, more time for YouTube videos.
One night, she confessed her cheating to a friend she played against. Her friend confessed the same thing. Both had been using cheat sites, both had been emulating the machine. They realized that they were just the meat between two competing algorithms; two algorithms which were much, much better at Scrabble than they would ever be. The sports players of the future are meat puppets, grinding out the space between human capacity and the unattainable perfection of the machine.
Or so we can hope. It’s unlikely such a situation would last for very long, as sports like all other industries must follow the dictates of the the economy. The primary cost in service industries is energy, which is subject to all kinds of natural limits, as well as artificial manipulations, from shale fracking to trade embargoes. Luckily, energy isn’t such a feature of sporting markets–although the Japan 2022 bid managed to address that also, by suggesting that its World Cup would be fueled exclusively by solar power, and a new kind of renewable energy harvested from the footsteps of visiting fans.
The secondary cost is labor–and so the biggest, tastiest thing to cut is the workforce. The world’s highest paid athletes banked a cumulative $3.15 billion in 2015/2016, most of it in the form of salaries supercharged by television revenues–money which could happily stay with their employers if alternative arrangements could be found, as well as opening up lucrative new forms of merchandising.
The solution, of course, is to do away with the meat. Automation is the order of the day. When Oxford University and Deloitte compiled a list in 2015 of the professions most at threat of automation, sports players came in at 227 out of 366–slightly better than hairdressers, but worse than airline pilots–with a 28.3 percent probability of replacement by machines in the next 20 years. But that’s still pretty high, 20 years isn’t very long, and, given the explosive growth of visual and computational technologies, it seems like a bit of wishful thinking.
We don’t need to get hung up on flesh here; in the realm of entertainment–which contemporary sport has been annexed by–virtual stars are doing well. Holograms have appeared on the stage at Glastonbury and Coachella–mostly notably in the form of the revivified ghost of Tupac Shakur–while Narendra Modi, the Prime Minister of India, has used the technology to appear at dozens of political rallies simultaneously. Hatsune Miku, Japan’s most successful virtual pop star, has played sold-out concerts across the globe. The last of these examples provides at least some grain of hope for those who still aspire to sporting glory: Miku is an open-source character, whose thousands of songs are created by an open, participatory community, creating a kind of crowdsourced, distributed fame. But the human sports idol of the future still looks more like world class e-sports player Saahil Arora than Cristiano Ronaldo.
In Homo Ludens, his classic 1938 study of play in human culture and society, the Dutch historian and cultural theorist Johan Huizinga repeatedly emphasizes the ritual nature of play, and its relationship to spaces of prayer and divination:
“The magician, the augur, the sacrificer begins his work by circumscribing his sacred space. Sacrament and mystery presuppose a hallowed spot. Formally speaking, there is no distinction whatever between marking out a space for a sacred purpose and marking it out for purposes of sheer play. The turf, the tennis-court, the chess board and pavement-hopscotch cannot formally be distinguished from the temple or the magic circle.”
In our increasingly alienated and virtualized civilization, what other beings come to colonize the circle? What is the field of play, the magic circle, protecting us from, and what is it summoning?
Through the lens of ball-tracking technologies, strategy analytics, instant all-angle virtual replays, and efficiency-optimized outcome, we glimpse the inevitable fate of a society obsessed with data, surveillance, and trade. The players training themselves against algorithmic better selves are the overpaid analogues of precarious pick-and-pack workers, following the dictates of wrist-mounted computers to the next, most efficient location in the warehouse, only to be replaced by actual machines with the next fall in robot manufacturing costs.
Sports, the most powerful of contemporary, globalized rituals, is calling into being a future that nobody wants but everyone keeps building anyway: a future based on automation rather than practice, and prediction rather than action. A world of most control and least chance; of minimum surprise and maximum profit; in short, all of the things that sport–that most marvelous and shocking endeavor–is supposed to stand against. And we do it because we can, because we perceive it as inevitable.
We are already at play on the field of ghosts. What we choose to conjure up with our magical technologies will decide if we get to play much longer.
How We Get To Next was a magazine that explored the future of science, technology, and culture from 2014 to 2019. This article is part of our Playing the Field section, which examines how innovations in sports affect the wider world. Click the logo to read more.
