Virtual production and LED volumes

Virtual production uses real-time computer graphics and large LED screens to place actors and products inside digital environments while the camera rolls. Instead of filming in front of green screen and adding backgrounds months later, the crew sees the final world on set, the actors react to it and the screens light the scene with realistic reflections and colour. Commercials and music videos adopted these methods quickly because they suit short shoots with many looks: a car can drive through five cities in one day, a product can sit on a mountain top and then on a beach, and a singer can perform in an impossible landscape. This lesson explains how LED volumes work, the role of real-time engines and camera tracking, how to light and expose with LED walls, the technical limits such as moiré and frustum, and how to plan a virtual production shoot.

What an LED volume is

An LED volume is a stage surrounded by large walls of LED panels, often curved, sometimes with an LED ceiling. The panels display images: pre-recorded video plates, still images or real-time 3-D environments rendered by a game engine. The camera films actors and props in front of the panels, capturing the background in camera.

• The main wall is the background seen by the camera.
• Side walls and ceilings provide reflections and light, even when not in frame.
• Floor and set pieces blend the physical foreground into the digital background.
• Separate LED panels on stands or wheels act as movable light sources showing specific images.

Volumes range from huge permanent stages for features and series to small portable walls for commercials, music videos and product work. Many rental companies offer pop-up volumes for a day or two.

Real-time engines and camera tracking

When the background is a 3-D environment, a real-time engine renders it from the camera's point of view. To do this, the system must know exactly where the camera is and where it points, so the perspective on the wall matches the camera's movement.

• Camera tracking systems use infrared markers, sensors or encoders to report the camera's position and rotation many times per second.
• Lens data, such as focal length, focus and iris, are sent to the engine so the virtual depth of field and distortion match.
• The engine renders the "inner frustum", the area of the wall the camera sees, at high quality and with correct perspective.
• The rest of the wall, the "outer field", shows a lower-detail version that provides light and reflections.
• Latency between camera movement and the wall update must be small; fast camera moves can reveal delays.

When the camera moves, the inner frustum moves with it, creating parallax: nearby virtual objects shift more than distant ones, as they would in real life. This is what makes the illusion convincing.

Lighting with LED walls

LED walls emit light, so they act as huge soft sources. A sunset environment lights the actor with warm light; a neon city reflects coloured highlights on a car. This interactive light is one of the biggest advantages over green screen.

However, LED walls are not perfect light sources. Most use red, green and blue emitters with narrow spectral peaks. Skin tones and some colours may look different under this light than under broad-spectrum sources. Many productions add traditional fixtures or LED lights with wider spectra to light faces and products accurately, while letting the walls provide ambience and reflections.

• Use the wall for reflections, edges and ambience.
• Add key light from fixtures with good colour rendering for faces and products.
• Match the colour temperature and direction of added light to the virtual environment.
• Control spill from fixtures onto the wall, which can wash out the black levels of the background.
• Program DMX-controlled lights to follow changes in the environment, such as passing streetlights in a driving scene.

Exposure, brightness and the camera

LED walls have a maximum brightness, measured in nits (candelas per square metre). The cinematographer must balance the brightness of the wall with the foreground lighting and the camera's exposure.

Camera settings must also be matched to the wall. LED panels refresh their images at a certain rate and are driven by processors that must be synchronized with the camera. If the shutter is not synchronized, the image may show flicker, rolling bands or scan lines. Shooting at frame rates and shutter angles approved by the LED technician avoids these problems; high-speed shooting with LED walls requires panels and processors designed for it.

Moiré, pixel pitch and focus

LED panels are made of tiny emitters arranged in a grid. The distance between them is the pixel pitch, measured in millimetres. When the camera focuses on the wall, the grid can interfere with the sensor's pixel grid, producing moiré, a shimmering pattern.

• Keep the camera at a sufficient distance from the wall relative to the pixel pitch.
• Keep the wall out of focus by placing actors well in front of it and using shallower depth of field.
• Avoid long lenses focused near the wall.
• Test the specific camera, lens and wall combination for moiré before the shoot.

A common rule of thumb is that the minimum comfortable viewing or camera distance in metres is roughly equal to the pixel pitch in millimetres, multiplied by a factor that depends on the camera and lens. A 2.6 mm pitch wall might need the camera several metres away to avoid moiré, but testing is essential.

Planning a virtual production shoot

Virtual production moves many decisions earlier. Environments must be built, tested and approved before the shoot, so the art department, visual effects team and cinematographer collaborate in pre-production.

• Choose between 2-D plates (video or photos) and 3-D environments. Plates are cheaper and fast; 3-D environments allow parallax and camera freedom.
• Hold a virtual scout in the environment using a headset or monitor to plan camera positions and lenses.
• Build physical foreground elements, such as floors, props and set pieces, that blend into the digital background.
• Plan lighting so that physical fixtures and the wall work together.
• Test camera, lenses, shutter and frame rates with the wall for flicker and moiré.
• Schedule time for the "brain bar", the team running the engine, to adjust environments on the day.

Virtual production in commercials and music videos

Commercials benefit from virtual production when they need many environments, controlled reflections on products and cars, or repeatable conditions for client approvals. Music videos use LED walls for impossible worlds and for lighting effects synchronized with the song. LED walls can also act as giant programmable lights, displaying abstract colour patterns rather than realistic environments. For small productions, a single LED wall behind a performer or product, showing simple moving images, can add a lot of production value at modest cost.

Crew roles and contracts

Virtual production brings new roles: virtual production supervisor, engine operators, tracking technicians, LED engineers and content creators. The cinematographer must communicate clearly with them, especially about lighting changes and camera moves. In both Canada and the United States, these roles are evolving under union agreements, and some are covered by technician or visual effects unions while others are freelance. Productions should clarify who is responsible for the wall's content, the engine and the safety of rigging large LED structures.

Content for the wall

The images on the wall must be prepared with the same care as a physical set. For 2-D plates, the content team shoots high-resolution footage with the right lens perspective, direction of light and time of day, often with multiple cameras to cover wide walls. For 3-D environments, artists build digital worlds in the real-time engine, optimized to render at the required frame rate. The cinematographer reviews content in advance for colour, brightness and contrast, because images that look good on a computer monitor may look different on a large LED wall seen through a camera. Colour management between the engine, the LED processor and the camera is essential; many productions calibrate the wall with the camera on a test day so that colours on the wall record accurately.

Blending the foreground and the wall

The most convincing virtual production shots hide the line between real and digital. Physical floors, props and set pieces extend into the virtual background, and their colours and textures are matched to it. Lighting on the foreground must match the direction and quality of light in the environment. Atmospheric effects such as haze can help blend the two, but haze also scatters light from the wall and reduces its contrast, so it must be used carefully.

When an LED volume is not the right choice

Virtual production is not always the best tool. Very wide shots may reveal the edges of the wall or require an enormous volume; fast camera moves can expose latency; scenes needing hard, direct sunlight are difficult because walls produce soft light; and small budgets may be better spent on a real location. The cinematographer weighs these limits honestly with the producer before committing.