Interview: Joe Letteri’s Visual Effects for The Hobbit

In December of 2011, I interviewed Joe Letteri for LivinginCinema and AwardsDaily. The four-time Oscar-winning visual effects artist took me deep into the process of crafting the characters and worlds of the Lord of the Rings films, King Kong, Avatar, and Rise of the Planet of the Apes. The main thing I took away from talking with Letteri was the extent to which he and the team at Weta Digital have redefined the very nature of visual effects on each of these films, and then incorporated the recent innovations into the next project while continuing to break new ground. After ten years, Letteri and the Weta team went back to Middle Earth with director Peter Jackson for a new trilogy of films based on J.R.R. Tolkien’s seminal fantasy novel “The Hobbit.” Much has changed in the world of visual effects since the Lord of the Rings films, and Letteri and the Weta crew reinvented Middle Earth using the all-encompassing nature of designing a world through visual effects a la Avatar while capturing Andy Serkis’ Gollum using performance capture in a manner perfected on Rise of the Planet of the Apes. The new challenges were doing all of that at 48 frames-per-second while also filming in 3D. After the release of The Hobbit: An Unexpected Journey,I spoke with Letteri again about his latest trip to Middle Earth, for which he has now been recognized with a seventh Oscar nomination for Best Visual Effects. Here’s what Letteri shared with me about visual effects in a post-Avatar world, making a film in 3D and 48 frames-per-second, and crafting The Hobbit: An Unexpected Journey.

Jackson Truax: In the span of just a decade or so since the Lord of the Rings trilogy was filmed, the art of filmmaking seems to have changed dramatically, at least when it relates to films like this and their cinematography and visual effects. What were the biggest changes you’ve seen that impacted The Hobbit: An Unexpected Journey, and how did those changes make designing the effects for the film different from the Lord of the Rings movies?

Joe Letteri: If you go back to the Lord of the Rings movies, a lot of what we were doing, effects-wise, was kind of in its infancy. It was really focused on creating a character, like Gollum, and trying to fit him into a scene. Or maybe, if you go back to Fellowship, creating matte paintings to spread out the environment… What’s really changed in the time since then, is that now, what we do for visual effects can encompass that whole range. So we still have shots where we’re just putting in a single character or a single creature. We have shots where maybe we were just placing the sky or doing a distant background. But in between that, we now have shots where can be creating the entire world. Everything that you see, including all the characters in it [have] become digital creations. That also means that you’ve got this whole process of visual effects starting much earlier in the filmmaking process. Now we start getting involved with pre-visualization. Instead of storyboarding, we’re making 3D models. And doing animation tests and trying to work out ideas for the film, oftentimes even before there are script pages written. And so visual effects has become embedded as part of the whole filmmaking process… You really saw that on The Hobbit in a couple of aspects. For example, Gollum. Whereas before, we weren’t sure how much of it was Andy [Serkis] just recording a voice and how much of his body we could use, and experimenting with motion capture… On The Hobbit, the very first scene we did was Andy in a motion capture suit, on the stage, with all the cameras and all the lights, something that was really unheard of ten years ago… So the technology really becomes embedded with the whole filmmaking process.

JT: You said in our interview last year that on Avatar you had to reimagine everything you knew because all of the sudden you were creating everything in a frame. So was The Hobbit the natural extension of that process?

Letteri: That’s exactly right. On Avatar, what we started to learn was in addition to the artistry, you really have to have a good foundation on the physicality of a scene. You need to know the science. The engineering, the math, the architecture, whatever is the physical counterpart to what you’re creating. It could be the hairstyling, or the costume design or the make-up. Any of those things all have digital equivalents… And on The Hobbit, that got spread across the whole spectrum of the shots that we needed to do.

JT:How did Peter Jackson’s decision to shoot the film in 3D 48 frames-per-second affect your work on the visual effects?

Letteri: What it meant for us was that we had twice as many frames to produce. All of our shots suddenly became twice as long. It wasn’t necessarily twice as much work all through the process. [Although] some parts of the process do require twice as much work. Anything that you’re doing by hand, frame by frame, painting out rigs or things like that…it does double the amount of effort. By the time you get to doing animation, some aspects of it [that] really take advantage of it, [such as] really fast movement like eye blinks, foot contacts, sword hits, things like that where you can be much more precise… That all works better for us now at 48 frames… Even though you can perceive more detail in the image, we already have all that detail going into the creatures and the characters and the elements that we create. So you’re just seeing more of what’s already been there.

JT:You said that on Rise of the Planet of the Apes, you did a lot of research studying apes and their movements. When you do a Lord of the Rings or The Hobbit film, is there a lot of research, from say things in Tolkien’s work? Or is the design of the effects more driven by the imaginations of Peter Jackson or the Weta team?

Letteri: Because we have a build-up of a lot of knowledge of what we’re creating anyway, most of the characters are derived in some way from being human. So whether it’s Gollum, or Azog, or the Goblin King or the goblins, those are extensions of human characters. So we can use what we know about humans and extend those. In the case of the Goblin King, we had to spend a little more time looking at masses of tissue and goiters and things like that. Because he’s got this disgusting growth all over him. It’s extended that way, based on the design ideas that Peter has. The specific new studies that we had to do for The Hobbit really came in the realm of the Eagles. We had done Gwaihir and the Eagles back on The Return of the King and actually even for Fellowship. But our knowledge was really limited. And we don’t see them as close-up as we do on this film. So this really was a whole new endeavor to really understand how eagles fly, how their feathers work. All that kind of detail.

JT:The way the book The Hobbit is remembered by a lot of people is that the story is very different from the feel of The Lord of the Rings. Both stories have a great sense of adventure and scope, but The Hobbit feels a little more whimsical, like more of a fairy tale or a bedtime story. Was that shift in tone something that you were conscious of when designing the effects?

Letteri: We were aware that his was meant to be a PG film. So especially in some of the fight scenes, you still needed to show them as big battles. But the cuts were a little bit [cleaner]. Everything was a little less brutal then it was in The Lord of the Rings. In [The Two Towers],you have that great scene of Boromir being shot to death by the Uruk-hai. There’s nothing like that here. So even though you know there’s a battle and heads are getting chopped off, it’s meant to be a little bit more, I guess lighter.

JT:When you go back almost a decade later to Rivendale and Hobbiton, was there any sort of cut-and-pasting of old places so you could focus energy on places the audiences hadn’t been before? Or even when revisiting previously-used locations, were you reimagining anything due to changes in the technology or story?

Letteri: We were reimagining everything. Hobbiton got rebuilt. Hobbition, what you see is pretty much all done live. That whole location was built. I think we replaced the skies in several of the shots. But we really didn’t change the look of it from the location that was constructed. And Rivendale, because a lot of it had been done originally as a matte painting anyway, there wasn’t really much that we could reuse, other than looking at the general feel and the general idea. There were some miniatures that were photographed for the original…matte painting. We dug those up and scanned them. Basically, they were just a guide… A lot of the buildings that you see are new. Because we’re seeing Rivendale from a different angle this time around. We had to basically build all new architecture but within the style of the earlier films.

JT: The signature character of motion capture is Gollum. In our last interview, you said on the Lord of the Rings films you had to have Andy Serkis go back and re-perform his movements on a motion-capture stage. In contrast, Rise of the Planet of the Apes was the first time you could record Andy’s movements on-set. How did you implement that technology on The Hobbit when revisiting Gollum after ten years?

Letteri: It was implemented on The Hobbit the same way it was on Rise of the Planet of the Apes. So when we came to The Hobbit, [Gollum] was the first scene that we did. It was a mature technology. Which was great, because that meant that it was one less thing for us to worry about. We could just focus on the performance aspects of it. In those first weeks of shooting, we were still shooting for the first time at 48 frames and [3D]. And really keeping an eye out for any issues that might arise because of those two things. Fortunately nothing did, but that’s where we were putting our focus. The motion capture worked really well for us… Technically, we knew we could get his performance. It was nice after ten years to come back to it and have it all work.

JT: When people think of The Hobbit, especially in relation to new characters, the first thing that comes to mind is the iconic Smaug. He appears in The Hobbit: An Unexpected Journey, but briefly, almost as a nod to the next two films. What was the biggest challenge in creating Smaug, and how did you overcome it?

Letteri: The biggest challenge…was trying to come up with an interesting design, but get something that would work in just the shots that were necessary. Because anytime you create a new character, you’re always looking to refine the character once you get through a few dozen shots. You really start to understand what the performance requirements are. That becomes a different kind of situation, when all you’re doing is making it perform in a few shots. It’s almost like when we saw Gollum back in Fellowship of the Ring.

JT: When you have a giant battle scene like the one in Goblin King Cavern, and you’re literally designing an entire army of goblins, what are the challenges there? Do you have to make every single goblin look unique? How do you accomplish that?

Letteri: You have to make them look unique enough that you can’t tell where you might be hiding duplicates. So what we do is we pick very distinct goblins with very unique features. And they become the basis for the ones that you really see. If you’ve got a close-up shot or a reaction shot, there’s about maybe fifteen to twenty of those that are unique. The rest of them, we have a way that we can extrapolate designs by mixing between those unique ones.

JT: When Peter Jackson announced that The Hobbit was going to be three films, not two, how did that impact what you were doing with the visual effects?

Letteri: It shifted things around a little bit. It meant that…some things that we thought were going to be in the first film were now getting moved to the second film. But it also meant that it opened up time to tell more of the story, especially the backstory…lot of the prologue, the destruction of Dale. At that point, a lot of that story, which we weren’t sure how much of was going to be the first film, got expanded.

JT: The Hobbit: An Unexpected Journey is already a worldwide runaway smash hit. Whether it’s from audiences or critics or industry peers, is there one piece of feedback you’ve gotten about the visual effects that’s been especially meaning to you?

Letteri: One of my favorite scenes, if not my favorite, in the film is still Gollum. And the feedback that I’ve been getting on how his character still really resonates with people. And I think how people are appreciating the differences in what we’ve been able to do…ten years on from the first time we saw him has really been great. Because we didn’t want to change the character, yet we knew how to do him completely differently from ten years ago. And the fact that people are recognizing that has been really great.