Eye tracking is changing the way we play video games

Edmund Huey is thought to have built the first eye-tracking device in 1898, consisting of contact lens with a hole in the middle. The lens connected to an aluminium pointer that showed the eye’s movement, but the device was so horrendous to wear that Huey gave subjects cocaine to ease their discomfort. Since then, eye-tracking devices have gradually become less intrusive, and have found uses in multiple fields from psychology to marketing. 

One recent application is the use of eye tracking in video gaming. The technology has a natural home in virtual reality (VR) headsets, and camera-based devices on the market can link with a PC to add eye-tracking capabilities to a number of different games. But now, bespoke devices are no longer even necessary: eye tracking can be achieved with any ordinary webcam or smartphone camera, opening possibilities for gaming integration even further.

One leader in the world of eye tracking for gaming is Tobii, based in Stockholm, Sweden. Thomas Papa, the director of product and business development for Tobii Gaming, previously worked in VR as founder and managing director of Mimicry Games in the Netherlands. “VR kind of plateaued, and I got a little bit bored from a tech perspective,” he says, explaining his change of direction. “I felt like the business around it was also stagnating, and I was looking for a new challenge.” 

For Papa, eye tracking is the next frontier in video gaming. “I've always been interested in how we can use tech to evolve the medium, and Tobii is very much at the forefront of that.”

How does eye tracking work?

There are many different methods for tracking eye movement, from specialised contact lenses to measuring changes in electric potential via electrodes around the eyes. “But the core technology that Tobii developed, and what it’s most famous for, is using a system that shines a pattern of near-infrared light onto a person's eyes, and then having sensors that look at the reflection of the pattern on the eyeball,” says Papa. “When you move your eyes around, that reflection changes, and the sensors are able to pick up on those changes in the reflection, also known as glints … It's not something you will notice, because it's in a different light spectrum, so it doesn't bother the user.”

Tobii’s gaming eye trackers, like the Tobii Eye Tracker 5, process these reflections via algorithms running on a dedicated ASIC chip (the same kind used in Bitcoin mining). “It's a specifically designed chip that's able to solve certain algorithmic challenges,” says Papa. “The chip outputs a variety of different signals that can then be used in the game. One of those signals is a gaze signal, so it's essentially an x, y coordinate of where the person is focusing on the screen, and it takes into account the screen size.” In addition to working out which point the person is looking at on the screen, the eye tracker can work out the position of the person’s head in 3D space, tracking pitch, roll and yaw. 

The Eye Tracker 5 is one of several Tobii products that work alongside a screen, but eye trackers come in various different shapes and sizes. “We have a wearable unit, which is a pair of glasses that has the sensors built into the lens,” notes Papa, but there are also high-frequency devices going up to 1.200 Hertz, typically used in research. “These are able to get even more granular data [on] things like pupil dilation or micro saccades, which are small movements that your eyes make when they're scanning an environment. [They] can tell us a lot about the user: it's almost like a fingerprint.”

A wide range of applications

Such devices have found various uses in behavioural science, education and even in medical applications; indeed, researchers have investigated whether eye tracking can be used to diagnose Parkinson’s disease. Papa notes that one Tobii customer in the automotive sector uses glasses-based eye trackers for training and assessment, with the in-built camera monitoring the subject’s point of view. “They’re using it to assess the skill of a new employee versus a very experienced employee in the process of, for example, putting together a machine,” he says. 

Tobii also worked on incorporating eye tracking into brain–computer interfaces (BCIs), which themselves have a rich future in gaming. The company initially partnered with OpenBCI and Valve to develop the Galea BCI/VR headset, though the final product ended up using Finnish firm Varjo’s technology. 

Papa says that BCI benefits from a multi-modal approach that incorporates various inputs, including heart rate, galvanic skin response and electroencephalograms (EEGs) in addition to eye tracking. “All of the different signals provide unique information, but they really shine when you pair them up … You need all of them in order to really get a good signal to be able to create software or games or experiences.” 

He adds that eye tracking is particularly important for BCI: “I would say it's a key one, because it's … directly connected to the brain.” 

How eye tracking is used in video games

There are many different eye-tracking companies, but few target the video gaming sector. TrackIR from the Oregon-based company NaturalPoint is one major brand in this field, along with Tobii’s range of gaming-focused devices. “I think we've struck gold right now in a niche that is fairly large,” says Papa of Tobii’s efforts in gaming. “It’s an underestimated niche, and also a niche that not a lot of people understand.” 

Tobii’s gaming-focused output is perhaps most well known in the VR space, since the company’s eye-tracking technology is integrated into VR headsets such as Sony’s PlayStation VR2 and the HTC Vive Pro Eye. In the Eye Tracker 5, Tobii also has a screen-based device that works with PCs, and the company partnered with a wide variety of game studios to enhance more than 170 games with eye-tracking features. 

But what can eye tracking actually add to games? Papa says when he first spoke to game designers, many suggested it would be natural to shoot lasers from your eyes, or pick up items or press buttons by simply looking at them. But in practice, this doesn’t work very well. “Our hands and our fingers are much better at controlling buttons and triggering actions,” he says. 

Instead, eye tracking has found a natural fit as a way to select or lock on to objects. A great example is the shooter game Rez Infinite for PSVR2, wherein which players can tag enemies by simply looking at them, then press a button to fire. It’s similar for the space combat game Star Citizen. “So in a dogfight in space, you can look at enemy ships and lock onto them using your gaze, rather than cycling through these 10 fighters that are around you,” says Papa.

Beyond locking on, eye tracking has been added to other games in various ways. In Shadow of the Tomb Raider, objects you can interact with are highlighted with a glow when the player looks at them, while in Ghost Recon: Breakpoint, the player can aim their gun along their gaze. In Dying Light, the flashlight can be directed with the player’s eyes, and in Desperados III, the enemies’ cones of vision are revealed when your gaze falls on them. Eye tracking can even be used to declutter the display, so that user interface elements like a health bar or mini-map only appear when the player looks at certain parts of the screen.

But one of the most popular features involves head tracking rather than eye tracking, to change the position of the game’s camera. In games like Star Citizen, Microsoft Flight Simulator and various racing games, the player can turn their head to move the game’s first-person viewpoint slightly to the left or right, or upwards or downwards, providing a natural and immersive way to look around the game’s world.

Expanding the market for eye tracking

All these features are clever and useful, but it’s been a challenge to get to this point, says Papa. There was a long process of trying to convince developers to take the time and expense to add eye-tracking functionality to their games. “We had to create an ecosystem,” he says, likening it to when a games console first launches and the platform holder races to convince studios to support it with a diverse range of games. But he thinks Tobii reached a tipping point about three years ago, “where instead of us talking to studios and knocking on the door saying, ‘Hey, look at this tech, look at all the cool things you can do with it’, now the studios are reaching out to us.” 

He adds that in some cases, some gaming communities demanded it. Fans of Microsoft Flight Simulator asked for eye-tracking support on the game’s forum. “And then the studio contacts us to say, ‘Hey, our community would like to have support for this tech, how can we make it happen?’”

Yet eye tracking still remains niche. The market size is necessarily limited by the need for a specialised device, whether it’s a $550 PSVR2 or a $299 Tobii Eye Tracker 5. Star Citizen players might be happy to spend big on their chosen hobby, but most people are likely to think twice before spending several hundred dollars to enter the eye tracking world. “Obviously, we need to bring the cost down of this type of technology, and thereby you broaden the market to make it more accessible,” says Papa. “We've been thinking about how we can leverage regular sensors, like webcams that are built into laptops.” 

Tobii already offers Tobii Horizon, which can track head movements via a regular webcam, and recently the company introduced Tobii Nexus, which uses AI to enable eye tracking with a normal camera. Papa says that this technology has been made possible by the algorithms Tobii has been training and the data it has gathered over the past 20 years. 

“We have trained an AI inference solution with a built-in geometrical model that solves population challenges using the minimum computational footprint,” he says. Now that eye tracking is possible without a dedicated device, “we're going to see a lot of new experiences and devices come out into the market.”

Tobii is currently focusing Nexus only on health care and education, although Papa notes the company is “exploring other areas.” A rival company already offers eye-tracking-enabled gaming using a webcam. The Beam Eye Tracker promises support for more than 200 games using regular webcams, and there’s an app to enable the technology to be used with smartphones. The company promises “eye tracking performance comparable with high-end hardware devices,” and though reviews of the Beam Eye Tracker note its performance isn’t as smooth as high-end devices like those from Tobii or TrackIR, it’s not far off. At just under $30, it’s an order of magnitude cheaper.

With a bigger prospective market, game makers have a greater incentive to add eye-tracking features to their titles, possibly making it a standard feature. There’s an opportunity for smartphone makers to integrate eye tracking into handsets, meaning we could soon see a new generation of mobile games that take advantage of the opportunities eye tracking offers. Even the next generation of consoles from Sony and Microsoft could integrate eye tracking as a way to differentiate offerings. 

The future of eye tracking

Aside from flight or combat simulations and racing games, where eye and head tracking offer a distinct advantage as well as a large bump in immersion, the reasons for adding eye tracking to other genres, like third-person action adventures, might seem less compelling. 

Maybe we’re not thinking big enough.

Papa says we’re used to seeing characters controlled by AI algorithms in games, and worlds governed or created by AI. He thinks we could see a third kind of AI system: director AI. He imagines an overarching game AI that “combines all of the data available in order to create the type of game experience that the players have by dynamically adjusting the flow of enemies, or maybe lighting or weather situations, to create that level of tension that the game is after.” 

It’s an interesting idea. Imagine that the game knows you’re looking intently at a particular door, enemy or area of the game world, so it decides to make something happen there; conversely, it springs a surprise away from where you’re looking. Valve’s multiplayer zombie shooter Left 4 Dead had a similar kind of director AI back in 2008, where the game adjusted the spawning of enemies and various other effects according to the specific situation of each player.

“AI will play more and more of a role in games,” continues Papa. “It's already a big part of games, of course, when you look at NPCs [non-player characters] and other types of systems. But the more an AI understands about users’ attention and state of being, the more advanced things it can do.” 

Making direct eye contact with NPCs is one intriguing possibility. Papa imagines a scenario where you play a hitman walking into a nightclub. “In your view cone there might be 50 people, but the gentleman in the back is actually able to understand you're looking at him, and if he's a guard, that could trigger all sorts of scenarios.” 

Getting to this future, where the design of video games incorporates eye tracking from the off, will require the convergence of numerous factors, such as usability, performance, price, availability and compatibility. The PC gaming landscape is fragmented, with some players boasting top-of-the-line hardware while others might not even own a budget webcam, so aligning all these factors will require collaboration between many systems and companies to make eye tracking affordable, accessible and essential. 

“It’s not just up to Tobii, and us building the right hardware that can output the right signals and is easy for devs to implement,” says Papa. “It’s also up to the game studios, the platform holders, other hardware vendors, the other members of the gaming ecosystem, to ensure that their systems are able to take advantage of it. And when they do, oh boy, when those factors are aligned, the experience is simply glorious. It’s amazing. And it’s something you don’t want to play without—ever.”