Film directors, cinematographers and visual effects artists all share the same goal: immersing audiences so deeply into the world of a film they’re watching that they believe everything is real. As long as the movie is running, dinosaurs really walk through the jungles of Kauai, apes run across the Golden Gate Bridge, animals can talk, and strange humanoid creatures live on other planets reached by spaceships traveling with warp speed.
For many years, adding creatures, explosions, spaceships, buildings, and anything else a director dreams up to a live-action film meant doing so in post-production — after the film was shot and edited. Visual effects artists became adept at matching the live-action lighting and, over time, rendering technology became sophisticated and computers fast enough to help them by mimicking the physics of light.
Soon, post-production visual effects began moving upstream. Previs, a roadmap that helped directors and vfx studios visualize (before production) the effects studios would create in post-production, became ubiquitous. Then, previs moved into “postvis” to help directors, cinematographers, and vfx studios visualize visual effects during the filming process. At the same time, filmmakers discovered virtual cinematography, VR, and AR. Cinematographers and directors could see visual effects roughly composited in real time through the camera lens during production. On location, they could visualize CG creatures superimposed into an environment on cell phones and iPads as they scouted a location. Filmmakers began looking through virtual reality headsets to view digital environments and characters. This virtual cinematography became so sophisticated that the digital environments and characters in The Lion King were shot in VR, and then refined in post-production. With each stage of evolution, the possibility of filming visual effects in camera, rather than adding visual effects later, had come closer to reality.
Filming visual effects displayed on LED walls, tested sparingly for a few productions, showed promise. And then Disney/Lucasfilm’s The Mandalorian episodic television production proved the technique could work on a much larger scale. By blending state-of-the-art LED and game-engine technology, Industrial Light & Magic, Epic, and other partners created a stunning, magnificent production stage. They call it a volume. LEDs on 20-feet high walls encircle a 75-foot diameter performance stage and cover the ceiling, leaving only a small opening for the crew to move props in and out and for the director, cinematographer, camera operators, and actors to enter. Practical set pieces designed to blend seamlessly with the displayed graphics blurred any distinction between real and digital. ILM calls the system, which works with Epic’s Unreal platform, “StageCraft.”
Three-time Oscar-winning visual effects supervisor Rob Legato, who supervised the Oscar-nominated visual effects for The Lion King (but did not work on The Mandalorian) describes the advantages of LED walls.
“It’s a wall, but what you feed it is a 3D environment,” he says. “The environment is based on the camera’s point of view; Unreal feeds the background as the foreground camera is tracked. So, if I pan right, it pans right. If I do a dolly move, it looks like a typical move in a 3D environment. It’s like the screen has disappeared and you’re doing a live composite. It’s a clever use of technology. The pandemic hastened its acceptance because you can build any set any time. Once [the LED stage] is built and working, it becomes inexpensive because you can shoot many things in a day. You can shoot in Grand Central Station and, 10 minutes later, be in Paris.”
Epic Games’ Miles Perkins, who is helping bring real-time workflows into the film and television industry, believes that the LED walls represent a fundamental shift in filmmaking as powerful as was Jurassic Park, which convinced filmmakers they could and should create digital characters.
“There’s a convergence happening,” he says. “The gaming community knows how to use the game engine for rendering in real time and for creating rules of engagement. People in visualization know the picture part. So all of a sudden there’s a creative sandbox, not just relegated to visual effects. Set designers and costume designers can bring things to the set that will be in the physical space next to something in the virtual world. The director of photography can capture both in the camera and they’ll feel like they’re in the same space. The DP can light a CG environment on set. Previs, techvis, postvis all led up to this. The difference is that hardware caught up.”
When filmmakers work in an LED volume, they capture the digital environment in camera. They see those digital environments, which can include landscapes or set extensions, displayed on the wall and film them. Light from the wall illuminates the actors, and images on the wall are reflected in surfaces on set. The director knows immediately if the visual effects will work because they see them right there.
But there are gotchas. One limiting factor is that the LED walls display two-dimensional images, meaning the system must track the camera to render an image so that they don’t look flat, even though they are flat. The DP can’t focus on the LEDs themselves without breaking the illusion. To avoid moiré patterns, the digital environments are often rendered with a soft focus. And there can be chromatic aberrations when shooting from the side.
So where might the next order of magnitude in compute processing and streaming take virtual production? The answer could be true 3D. That is, light fields — three-dimensional images created digitally or captured with light field cameras and displayed with light field displays. If light field display (LFD) walls were to replace LED walls, they could solve some problems with the LED walls, offer all the advantages of the LED volumes, actually immerse filmmakers in a virtual world, and open the door to the future of entertainment.
“Light fields are the inevitable ultimate result of our effort to virtualize reality,” says Scot Barbour, an independent filmmaker and former VP of production technology at Sony Pictures Entertainment who has been working with virtual production stages for more than a decade. “Where we are with LED walls is kind of like where video games were with Pong. That difference between Pong and triple-A games is where light fields can take us. If you’re wearing a headset and interacting in VR, that’s maybe halfway to what you could do in a light-field volume, and you could do it without any apparatus. You would have full six degrees of freedom, a full holographic image. It would feel real because to your eyes it is real. The game engines with real-time ray tracing will allow interaction with light fields, and light field displays will be the visualization mechanism. If you can synthesize light, there’s no greater realism, period.”
As for the problem-solving? A DP could shoot without concern about moiré or chromatic aberrations, and without focusing on an LED screen. “When you’re no longer focusing on the display but on the light, there would be a fundamental shift,” Perkins says. Rather than a visual effects artist compositing a dinosaur into a film in post-production, an actor could stand on stage in front of a T. rex — a digital T. rex created some time earlier by an artist.
“On an LED stage, we try to hack our way around the problem that you still have a flat screen,” says Magnopus co-founder Ben Grossman, who also received an Oscar nomination for visual effects and virtual production in The Lion King. “In theory you wouldn’t have to do that with a light-field display because each person in the room would see the content in their perspective. For every point on the screen, you’d see what’s appropriate from that angle, whether taller or shorter. In theory, we wouldn’t be able to perceive a screen at all. It would more closely emulate the photons of light in the real world.”
“Imagine if there were no disbelief,” Grossman continues. “In theory, that’s what light field displays can do. You’d have no reason to believe what you’re seeing isn’t real.”
The question is when.
“There are a lot of things that need to come into play,” Perkins says. “Image fidelity, color accuracy, the ability for a DP to light in specific ways. But with that said, I think we’re on the precipice of a revolution.”
Data is a gating factor. “A light field is kind of like all the available possible viewpoints for a particular location,” Grossman says. “If you think about rendering and streaming all those things to all the possible views that exist, it’s a lot of data. And to do that for an area in motion would be an insane amount of data. But we could do small areas sooner, say everything outside the windows on a set.”
Barbour imagines some of the first uses for light-field display walls would be for blocking – to put digital characters into the set with actors, even if the characters would be recreated later.
Perkins imagines using light field displays as elements in an LED volume. “You could have LFD things in the foreground where you don’t want to worry about the focal point,” he says.
For all these people, the promise of light-field displays goes far beyond its potential use in virtual production.
“Entertainment is not static,” Perkins says. “I think we’re going through another revolution, and display technology is part of that.”
Says Grossman, “The real value of light field displays is that if you’re going to make a film, once you have built up the light-field content, why not put LFD screens in people’s homes so they can experience the content in the same way as the way it was produced? To me, the idea of giving someone a 3D world at home is more interesting than giving someone a 2D facsimile of an amazing 3D world. Just giving the consumer a flat thing doesn’t realize the full potential of light-field displays.”
Someday the light emitted from your display will put dinosaurs, full-sized apes, talking animals, strange humanoid creatures from another planet, and perhaps even the deck of a spaceship, all filmed on an LFD stage, right in your living room. And you will believe they are real.