Performance capture injects emotion into virtual worlds

James Cameron, the director of some of the film industry's biggest blockbusters, quipped in 2009 that, "Actors don't do motion; they do emotion." He was at San Diego's Comicon at the time, talking about his then yet-to-be-released film Avatar and how the technological process known as performance capture was going to revolutionize filmmaking.

The process involves electronically tracking an actor's movements and facial expressions and using that data to animate computer-generated (CG) characters in order to make them more lifelike. Strategically placed sensors on the actor's body allow cameras to digitally record the precise angles, speed and trajectory of each movement. These are then translated into motion sequences and mapped onto 3D models of CG characters using animation software.

Performance capture empowers actors, Cameron said in San Diego, because it frees them from their physical body while preserving every moment of an actor's performance on set.

Peter Jackson, sitting beside Cameron on the same panel discussion, agreed. "It's an extension of the makeup process," said the director of The Lord of the Rings trilogy. "It's not hugely different than what Lon Chaney did." 

(Chaney was an actor in the 1920s known as the "the man with 1,000 faces" on account of his use of elaborate make-up to create characters such as the Phantom of the Opera and the Hunchback of Notre Dame.)

Jackson would know, having outfitted British actor Andy Serkis with a bodysuit covered in an array of motion sensors in order to capture the movements for his CG character, Gollum, in the Rings trilogy. 

While motion capture records only body movements, performance capture takes in all elements of an actor's performance — body movements, facial expressions and voice — simultaneously. It makes for more convincing and realistic characters, said Tameem Antoniades, the head of creative at the U.K.-based video game developer Ninja Theory and designer of the games Enslaved: Odyssey to the West and Heavenly Sword. 

"So much of our communication is non-verbal," he explained. "We can tell whether you're lying or whether I'm telling the truth just by looking at each other's eyes and faces."

Heavenly Sword (2007), which starred Serkis and Anna Torv (Fringe ), was the first video game in which performance capture was used. Antoniades admits that during the creation of the game, the developers weren't sure what the end product would look like because the final animations that use motion capture take up to a year to create.

"We were totally working blind," said Antoniades, "but I think the results were better than we expected. It proved to us that you can have subtlety and emotion, and you can engage players emotionally through the expressions of CG characters."

Serkis directed the actors on Heavenly Sword, an action adventure in which gamers take on the role of Nariko, a warrior princess who protects a sword imbued with great power. He and Antoniades co-directed the performance capture on 2010's Enslaved, which starred Serkis as Monkey and Lindsey Shaw as Trip in a tale based on the 16th-century Chinese novel Journey to the West.

The technology used to capture actors' performances didn't change much in the three years between Heavenly Sword and Enslaved, said Antoniades, but there was a creative shift.

"We knew you could do expressions realistically," he said.

So, while the actors gave exaggerated, theatre-like performances in Heavenly Sword, on the Enslaved set, they played things straighter and more natural.

"Anything you can do in a movie you can do in a video game in terms of acting performance and subtlety," said Antoniades.

Fight Night Champion, which California-based developer Electronic Arts released on March 1, makes use of performance capture to inject a movie-like narrative into what would otherwise be a pure sports simulation. The game stars LaMonica Garrett as Andre Bishop, a young boxer struggling to become a champion whose character gamers take on, and Eliza Dushku (Dollhouse, Buffy the Vampire Slayer), who plays a daughter of a fight promoter.

Brian Hayes, creative director and lead producer of the game, was responsible for overseeing the script and cinematics for the story mode. He said performance capture was essential to getting the level of quality the developers wanted in the animations. There's no point in creating interesting characters and environments if those characters are just going to stand there like puppets with their mouths moving up and down, he explained. Performance capture helped deliver compelling scenes.

"It makes it seem much more like you are engrossed in a real movie," he said.

When real people talk, they use more than their mouths. They wrinkle their brows, wave their hands around, hunch their shoulders — and it all happens at the same time. In creating a cinematic scene with dialogue and character interaction, being able to record body and facial movements together is essential, Hayes said.

"It means that when we recreate the scene in the game, everything looks human," he said.

Filmmakers and video game creators are using the exact same method to tell stories, says Antoniades. Cameron, Jackson and Steven Spielberg all visited the Heavenly Sword and Enslaved sets, just as Antoniades visited the Tintin and Rise of the Apes film sets. They compared notes on how they were using performance capture and what they were doing with the data afterward.

"Everyone's going about it in their own particular way," said Antoniades. "Filmmakers go about it in a way that is much more similar to traditional filmmaking. We don't know anything about filmmaking, so we do it in a totally different way, but it's still the same result. It's trying to tell stories in a virtual way."

Having actors employ their craft in the process of performance capture ensures that audiences can't help but connect with their digital characters on an emotional level — and that's true of films and games, Antoniades said.