Avatar - on the Cutting Edge

The director of Terminator and Titanic explains how movies will be transformed by motion-tracking and 3D technology

Three-time Academy Award-winning director James Cameron is a pioneer in the field of motion capture. In the mid-'90s he used the nascent technology to create the massive crowd scenes and stunts in his blockbuster Titanic. These days he's still at the cutting edge of the technology, but he prefers to call motion capture "performance capture" because, as he points out, "actors don't do motion, they do emotion."

Cameron is in the midst of his latest film project, Avatar, which is his most technologically innovative film to date. The futuristic movie about an ex-Marine will be released in 2009 simultaneously with a massive, multiplayer, video game based on the film.

Business Week couldn't catch up to Cameron for a sit-down interview, since he's busy creating Avatar, but reporter Aili Mc Connon was able to engage the director, via e-mail, in a discussion of how motion-capture technology has spurred innovation in cinema and made filmmaking more cost-effective. The following are excerpts from their virtual conversation:

What has motion capture meant to the film industry and to your work?

Performance capture (Perfcap) in recent years has enabled such stunning [computer generated] characters as Gollum (in Lord of the Rings parts 2 and 3), "King Kong," and Davy Jones (in Pirates of the Caribbean) to be brought to life. The technology is critical to the realization of my dream project, Avatar.

In fact, Avatar wasn't possible when it was first written 11 years ago, and only through pushing the technology to new levels over the past year and a half have we reached the point where the film is finally possible to make.

What innovations have you developed for Avatar?

We have greatly enhanced the size of the performance-capture stage, which we call The Volume, to six times the size previously used. And we have incorporated a real-time virtual camera, which allows me to direct [computer-generated] scenes as I would live-action scenes. I can see my actors performing as their characters, in real-time, and I can move my camera to adjust to their performances.

In addition, we have pioneered facial performance capture, in conjunction with our visual effects partner, Weta Digital. This technique eliminates hours in the makeup chair, and various other discomforts, for the actors. Previously, actors needed to have hundreds of tiny spherical markers glued to their faces, and they couldn't touch their own faces throughout the shooting day as a result. With the new system, a lightweight head-rig can be donned minutes before shooting.

We have had great success, and other filmmakers such as Steven Spielberg and Peter Jackson have worked on our virtual stage doing tests for their upcoming films, and given high praise to the system.

Does the rig cover the whole head, including your face? Does it capture fine facial movements?

The rig is a small skull cap, made from a cast of the actor's head, so that it fits comfortably while being tight enough to avoid shifting. It acts as a base for a strut which resembles a concert microphone (visualize Madonna in concert), except instead of a mike in front of the face, it has a tiny camera. The key to it is the software, which interprets the movement of the actor's face, pupils, and eyelid responses as the image flows in from the video feed of the head-rig camera.

In what directions do you see the technology going in the short term?

Improvements to the software and higher computation speeds and storage densities will enable us to have more realistic environments and more refined facial emotions and hand movements. Hand movement, for example, is still at a crude state.

On Avatar, we're working on-stage at a reality level equal to an '80s video game. At the end of the day, after a year and a half of post production, the images seen by audiences will be 100% photo-real, i.e. indistinguishable from photography. But for our day-to-day shooting, the image can be improved a lot.

Another area which needs improvement is the lighting. We need to improve its ability to handle cinematic lighting, the casting of shadows and so on. All of this can be improved as Moore's Law raises the speed of processing and as upgrades to the software become available.

In addition, we're developing ways for [computer-generated] characters to interact with actors who are being photographed on real, live-action set. We will have real-time stereo (three-dimensional stereoscopic, or 3D) composites of characters, which will be viewed by me in the eyepiece of the camera while I'm shooting a live-action scene. This will be revolutionary. We're not quite there yet, but we hope to have that by August, in time for our live-action shoot in Wellington, New Zealand.

Long term, what do you expect?

I expect that more filmmakers will embrace the technique and apply it to different types of scenarios. For the creation of fantasy and science-fiction characters, Perfcap will largely replace makeup and prosthetics.

Actors need not feel threatened by this change in technology. It doesn't replace acting, in fact it's designed to empower the acting and directing process, as opposed to the traditional [computer-generated] animation process, which uses only the actor's voice, and in which a committee of animators perform the character, operate the camera, and do the lighting.

I believe it will make fantasy filmmaking much more user-friendly for filmmakers, actors, and studios, and ultimately bring down costs. It's just now possible to create photo-real human [computer-generated] characters, but it isn't cost effective.

Many other fields, from medicine to automotive design, now use similar motion-capture systems (though on a smaller scale). Do you ever run across or dream up non-entertainment applications yourself?

I'm bore-sighted on the cinematic process. While one can generally imagine all the industrial and science applications, I'm not interested in developing them. However I can visualize a number of uses for the technique in advanced forms of entertainment, at theme parks and so on.

What role will 3D play in the future of film?

Here's what can happen, although it's too early to say if it will: 3D can become ubiquitous as digital cinema replaces film. As digital cinema rolls out, stereo follows—and in some cases leads the charge, as we have seen recently with the digital 3D releases of Chicken Little and Monster House forcing the installation of hundreds of new digital projectors.

There will eventually be major titles available from all studios at some screens in almost all multiplex cinemas worldwide. I would say the horizon for this is five years. 3D can become a fully accepted way in which audiences view movies. It will become another consumer choice, like premium or regular gas. The premium experience of 3D will be the preferred viewing experience for action, animated, fantasy, and science-fiction films.

3D's broad acceptance at theaters will generate enough content that consumer-electronics manufacturers will make home players and monitors available. The technology exists now, but is not readily available as off-the-shelf products. 3D display will become a must for video and computer games.

In 20 years, stereo media may become the preferred method for displaying all information, including news and other broadcast media. The density of information one can place on a small screen becomes much higher if it's stacked in three dimensions.

Is there something beyond 3D in film? Could we ever see in cinema the same kind of physical participation we're starting to see in video-game consoles like Nintendo's Wii?

Imagine a movie in which the viewer is swept along by a narrative, following the action from place to place, but without the intervention of a camera. You can choose which character to watch in a scene, as if you're an invisible witness standing there while a real event plays out. This is still years away, at a level of realism people would consider cinematic, but certainly not decades away.

I can imagine the dense fantasy worlds I like to create for movies having an equal or greater life in a world of interactive play, authored by others, in a partnership. Of course, add massive multiplayer capability to this, and people will never leave their homes.