Jackson Truax talks Rise of the Apes Visual Effects with Joe Letteri

Like countless other craftsmen in the film industry, Joe Letteri is far from a household name. Still, if you’ve appreciated the visual effects of the Lord of the Rings films, King Kong, Avatar, or this summer’s Rise of the Planet of the Apes, you’re familiar with Letteri’s work, and how he and the crew at Weta Digital have spent the past decade being great innovators in the world of visual effects, creating and redefining several times over the art of the motion capture performance. Letteri has already won four competitive Oscars and a technical achievement award, and is once again a frontrunner in this year’s race for his work on Apes. I recently sat down with Letteri, and he generously spent a half-an-hour taking me through the highlights of his career thus far, the new ground he and the Weta crew broke on Apes, and providing a glimpse into some future projects. Here are the highlights of our in-depth discussion:

Jackson Truax: Take us back to your early days working at Industrial Light and Magic. How did end up working there, and what did you gain from the experience?

Joe Letteri: I had been learning computer graphics on my own. I was really interested in doing organic effects, clouds and things like that… I got a job at ILM, and became interested in this whole side of “How do you make things look realistic?” The first [movie] I got to do anything on was…Star Trek VI… The opening shot…the planet explodes in this…big ring of fire. I got to use these ideas I was working with on natural looking clouds, and twist them around to make this natural-looking fire-ball explosion. That was my first shot in film… I just kept doing it from there. Our next film was Jurassic Park. The idea there was to make realistic dinosaurs… As I kid I used to always draw dinosaurs, and think about what they would really look like. So to have the chance to model that, and figure out what to they look like [and] how do they move, was also really a great experience. From that working with creatures, I started getting interested more and more in characters. We worked on Casper… [Here’s] a character who was a major element in the film, and a fully animated 3D character… It was a nice progression, to be able to figure out these elements…and building on it to create these characters.

JT: Through the progression of these films, you worked your way up from being a computer graphics artist to being a visual effects supervisor. What was your journey of climbing the ladder?

Letteri: It really has to do with…understanding what it is you’re trying to achieve, and how to produce those effects. It’s problem solving, not just in a technical sense, but also in a creative, filmmaking sense. Understanding how the work that you’re doing needs to fit into the narrative, the edit, the overall scenic composition of the film. Having that kind of a perspective means that you’re not working on just the small details. You’re looking at the big picture, while still having to make all the small details work. That’s pretty much what you do as an effects supervisor.

JT: You were the visual effects supervisor on the 1997 special edition of Star Wars. What kind of work were you doing on the film? And what were the challenges of trying to improve on what’s arguably the most iconic film of all time?

Letteri: George [Lucas] wanted to…bring Jabba the Hutt into the film. He didn’t have the ability to do that when he did the original… So the trick there was trying to take the Jabba the Hutt we knew from the third film, and create a character out of him that would move in three dimensions. Remember, from the third film, he was a big slug, he didn’t move… When [Lucas] shot the scene originally, it was done with an actor…a guy with a vest… We had to figure out how to get Jabba the Hutt to move where this guy was moving, because Harrison Ford’s eyeline would always be to his eyeline… And he had to move around a lot. So we got to actually redesign Jabba the Hutt… In the third film, he’s sitting with his tail off to the side. We had to put his tail behind him to he could move… When [Ford] walks around behind the guy, he’s walking a around human, but now Jabba’s got this…tail… We got [Ford] to lift a little bit to make it look like he was stepping. We basically reanimated [Ford’s] image. And then Jabba did a little blink when his tail got stepped on. That kind of sold the moment to put it all together.

JT: At what point did you decide to leave ILM and join Peter Jackson’s WETA Digital?

Letteri: I knew some people who were working on the first [Lord of the Rings] film. I knew that they needed someone…to start prepping for the second film… So I talked to them… The chance to work on Gollum was what really attracted me to want to go [to New Zealand] and take that on. Because it was the first time…that we were going to [create] a speaking character that was so human-like.

JT: WETA has really taken on a whole new life since then. How was it impacted in the long run by the work on the Lord of the Rings films?

Letteri: When I went down there to start prepping for Gollum, we really had to build up a new kind of infrastructure for doing character films… [There were] a lot of challenges that no one had ever really dealt with before… With us, the big breakthrough was coming up with a way to do subsurface scattering of computer graphics. We did that with Gollum for the first time. That gives you that translucent quality to the skin that really makes it look believable… It helps with the eyes. And then there we spent a lot of [time] understanding how faces work, and facial animation, and decoding that whole process. And then also muscle systems…and skeletons. Everything that goes into making a physical body work, and “How do you create that for a computer graphics character?” Having gone through all those changes and putting that all together, we started building a whole new infrastructure…that would allow us to take on more and more of this kind of work.

JT: You won your first two Oscars for two Lord of the Rings films, as well as a technical achievement Oscar for the third. What did that recognition from the Visual Effects Branch and the Academy mean to you?

Letteri: It means I didn’t screw it up too badly… When we were working on it, we had no idea what people were going to think. Because the techniques we were using were so new. [It] took so much artistic effort to…just get everything looking right… When the Academy recognized it, I was blown away. That was a really nice acknowledgement.

JT: After the success of the Lord of the Rings films, what was the process of making King Kong? It seemed that Peter Jackson and Co. had complete free reign to do whatever they wanted to do on a very effects-driven film. What were the biggest challenges and most exciting aspects of working on that kind of project?

Letteri: The biggest challenge for us was Kong himself. He’s the guy in the title. He’s a 25-foot ape. We wanted him to look realistic. We wanted the audience to feel empathy for him… To be a character that audiences [would] understand… He’s not a monster. He’s a big creature… Sometimes creatures can be scary…this is the last of his kind. This is guy that’s seen all the rest of his kind come and go, and that’s it… We all know how it’s going to end. But you still want to understand what he’s feeling as you’re going through all that. To have a character that has to do all that throughout the film and to not have any dialogue, it just meant we had to pay really particular attention to his facial expression, the detail in the face, especially the eyes. You’re really focused on all the minute clues to get a sense of what he’s thinking and how he’s going to react.

JT: Even two years after its release, it feels safe to say that Avatar is among the most influential visual effects movies of all time. What were the biggest challenges you had working on it? Did it feel groundbreaking while you were in the process of making it?

Letteri: We knew we were breaking a lot of new ground when we made it because the scope of the work was so big. We had to go back and…question everything we thought to be right. So much of what we do is based on…a lot of science. There’s a lot of physics. There’s a lot of biology that goes into it. You really have to understand the things you’re putting up on the screen in all of its aspects, whether it’s a creature, or a tree, or a building, or a cloud… Avatar had all those elements. It wasn’t like most films, where you’re just doing a little bit of something that you focus on. So we really had to look at everything and think about how all these would work in an integrated fashion. The addition of using 3-D photography also meant that some things that you used to do in visual effects, like shoot explosions on a stage and composite them in later, weren’t going to work. Because now we’re flying the camera around the explosions and they’re happening in stereo… We had to reevaluate all of our techniques, and really understand the physical basis for all of them. And come up with a way to create all of them. And do it in such a way that they would integrate together as if you were…in the real world and all these things were happening all around you.

JT: The same year Avatar was out you had worked on The Lovely Bones. The film and the special effects seemed caught in the debate of what was appropriate for a certain take on the material versus how showy and obvious the special effects should be. Was that something you thought at all about during the creation of the film? How did you find the voice of the effects?

Letteri: This…was kind of the fine line that we were walking. We wanted it to feel like these were things that were…coming out of Susie’s unconscious… They were things that she could relate to, but it wasn’t a world that she was just imagining. It wasn’t only a world that was coming out of her head… You couldn’t introduce things that were unfamiliar to her. But it wasn’t meant to be science-fiction. It was still meant to be her world. That was the balance we were trying to strike. That was the thing that felt like it would give the most emotional appeal to it. It would really resonate with her in a way that, sometimes would be scary, sometimes would be comforting, but would fit the arc of the story.

JT: Take me into the discussions of you first coming on to Rise of the Planet of the Apes.

Letteri: I was approached by John Kilkenny, who’s the VP of visual effects at FOX… We had just finished Avatar. [Kilkenny] had the script for Caesar: Rise of the Apes, as it was called at the time. He sent it to me and said, “You’ve got to read this…” It was just a great story. I thought, “We have to make this movie. So, what’s the plan?…” [Fox] had already been in discussion with [director] Rupert [Wyatt]. They were looking at just how we would put the thing together. And, “By the way, we would like it out next summer.” It was a pretty tight schedule… It had to be a summer release. It just had that feel… I immediately thought, “We want to do this with performance capture.” This really takes us…back to the way we did Gollum… The actors being in there, being apes with all the other actors. And then we paint them out later and we replace them. However…what we never got to do on Lord of the Rings was record those movements directly. [Andy Serkis] always had to go back and re-perform those movements on a motion capture stage, to match the movements that he did when he was acting with the other actors. We thought if we could really crack the problem of, “How can you make a motion capture set-up work on a live action stage? That would be ideal…” So we did that. We came up with a way to do that. That meant when you see a performance of Caesar, you’re seeing [Serkis’] performance as he did it in the moment with everyone else.

JT: What was the innovation that made the new way of shooting possible?

Letteri: When you’re doing motion capture normally, you’ve got these reflective markers that you’re gluing onto the suit… They’re little reflective balls, that basically velcro on to the suit. And they’re retro-reflective. So you’ve got cameras all around, looking for the movements of those dots and just isolating them. Those dots tell you essentially where all the joints are in the body. That’s what they track and they recreate the motion by tracking all those dots. If you have any kind of stray signal like bright highlights…a car with the sun reflected off a chrome bumper, it’ll confuse the system because it will think it’s another dot it’s supposed to be tracking and suddenly the character’s arm gets pulled over to where that dot was. So things like that you have to eliminate. Plus, the lights that you use to illuminate all those will interfere with the lights that the camera needs for the film… So the technique we came up with, was to replace all those retro-reflective markers with infrared LEDs, and they’re phased so that they’re off when the shutter’s open and they’re on when they shutter’s closed. So we still get our data, but the lighting doesn’t contaminate the camera, because it’s got an IR filter on it… We tuned it so that it wouldn’t affect the film camera, and it wouldn’t affect the other lighting…or even the sun wouldn’t affect what we were getting. So we could work in broad daylight, and around cars, and everything else. It worked perfectly for us… [Wyatt] filmed [the scenes] without [Serkis]. We called them “clean plates,” because you’ll need that to paint [Caesar in] later… We take measurements of everything. We photograph everything. Because [we] need to recreate it and match it. We built a computer reconstruction of every set that we’re in. We have to match it physically. And we have to match all the lighting… There will be questions that will come up about what might need to happen in a certain scene to get a bit of the action. Especially when you start to getting into things that involve stunts or things that the performers just can’t do. They might be able to do part of it. [If they’re] taking a big leap off of a building and then landing onto the street. We might get them to jump off of a box, and know that they’re going to jump just three feet down… Knowing that we can…animate the rest of it.

JT: You said on the Apes DVD that “Serkis brings a lot of experience to it that’s really relevant.” How did Serkis find and develop this style of acting, and how has it evolved throughout the films you’ve worked on?

Letteri: The way that [Serkis] approaches a character, he just goes into it a hundred percent. That’s really what you want. As an actor, [Serkis] is looking for what makes that character who he is… The reason that I think that works so well and that’s so important is…there’s no holding back. You’ve got this character that’s not going to look anything at all like you when you’re done. But it still has to carry that whole performance through. So everything that you do, you really kind of have to commit to it. You can’t rely on…the camera [seeing] who you are, and there might be some subtleties there that…you can get away with… Every movement has to be true.

JT: What’s your research process on a film like this? Did you spend a lot of time studying apes and how they look and move?

Letteri: We did. There’s a really nice size group of chimpanzees at the Wellington Zoo. The zoo was great. They gave us access to them. We spent a lot of time…filming…and photographing them. And getting behind the scenes with them, watching them when they do their grooming and their medical check-ups… A lot of the apes that you see in the movie…the ones that you see at the beginning of the film in the jungle, the background ones in the primate shelter, a lot of those you would recognize if you went to the Wellington Zoo. We photographed them so much, we just used them as models… We saw what they were doing and how they reacted to each other, and we took ideas for their behaviors for a lot of the action… There’s so much going on, even background action… when they’re sitting and doing things with each other, when Rocket’s there and the other chimps are grooming him… We’re looking at, “What do real chimps do in a situation like this?”

JT: Serkis said of his performance, “It’s not about servicing a visual effect.” Are motion capture performances visual effects? Are they performances? Or is it the ultimate achievement in both?

Letteri: I think it’s the ultimate achievement in both. If you’re really talking about a character that you can believe in on-screen, that requires a great performance… So if that’s just an actor, and you’re not talking about a CGI character, you’ve got a great performance… If you’ve got something that looks totally believable on-screen that you’ve never seen before, that’s a great visual effect. If you put the two together and you’ve got a character that you’ve never seen before, that can perform in a way that’s not just believable, but emotionally [compelling], to me that’s the ultimate achievement.

JT: Now that you’ve been shortlisted for another visual effects Oscar and the “Bake-Off” is on the horizon, can you share what footage you’re planning on sharing with the members of your branch, and what you might be saying about your work on this film?

Letteri: What that’s all about is highlighting the best pieces of the work. Generally what you’re trying to do there is just…encapsulate the arc of the story in a ten-minute format… [Using] some of those moments that give an overall sense of what the work is, what the achievement is, but also how it fits the story. In all of these visual effects discussions, everyone is always looking for, “Does it support the story? Does it support the film?…” If it takes you out of the movie, it’s kind of pointless. We’re hoping to show that…this actually works with the flow of the movie; in fact, it helps make the flow of the movie.

JT: What does this recognition from the Academy mean at this point in your career? For you personally, as well as what it says about the state of motion capture in film?

Letteri: It really comes down to attention to detail… There’s so much detail that goes into this, especially when you’re doing characters…if something is wrong, the audience knows that it’s wrong… To me it’s a validation that at least we’re looking at the kinds of details that are appropriate to really make a character believable.

JT: Looking ahead mere days, The Adventures of Tintin is coming out. What’s the process of doing these effects on an animated film? How is it different?

Letteri: It’s technically very much the same. The only difference is…what kind of information…you gather on the set. [The] additional photography and measuring you do, that goes away because you’re creating the world completely. Other than that, we’re working, like we did with Avatar…there’s a virtual camera. You can see the world. You can see the actors through the virtual camera. It’s the same performance capture system with markers on the suits… You’ve got the facial cameras set-up… What you have to pay particular attention to when you’re creating it, is that you don’t fall short of the mark that you’re trying to set. When you have a live action film, and you’re putting Caesar into a photograph, you’ve got everything around you in that picture that tells you what’s real or not. The quality of the light, all the detail that’s in the set. The other actors around, what you see in their eyes. And how their costume’s moving and how their hair looks. You have all the queues… [When making] an animated movie, it’s easy to convince yourself “that looks real” when it really doesn’t, because you don’t have anything to hold up against it. That’s the part that you really…have to stay focused on.

JT: Talk about returning with Peter Jackson to the Middle Earth on The Hobbit. Are you going back and recreating the world audiences fell in love with ten years ago? Or can audiences expect you to take us to a new place?

Letteri: It’ll be some of both. There’s places that we’ll revisit that were…obviously in The Lord of the Rings, Rivendell, Hobbiton. But there’s new places that we’ve never seen before, like Goblin King Cavern, and places like that.