One Set, Infinite Outcomes: How Virtual Studio Technology is Revolutionizing TV Production

What is virtual studio technology, and how is it changing broadcast pipelines for good? Find out in this interview!

As part of a new series of webinars, we brought industry experts from Jett Sets and Safe Frame together to talk through their virtual studio technology. Read on to learn the secrets to setting up a virtual studio, the benefit it brings to broadcast budgets, and more.

What is a virtual studio?

At its most basic level, a virtual studio is a stage that can seamlessly merge real actors or props with computer-generated characters, backgrounds, or effects. Sports broadcasters can use virtual studios to enhance stadiums with virtual crowds and billboards. News broadcasters can also use virtual studios to capture talent anywhere from ancient Rome to Mars, all in real time. The possibilities are endless!

“Everybody’s familiar with The Mandalorian and how they used Unreal Engine. Other shows are picking up on that and doing the same things,” says Joe Jett, CEO and VFX Supervisor at New York-based Jett Sets. “These shows are now constantly looking for talent. [That means there are] some pretty big opportunities in the next couple of years for jobs [in virtual studios], whether it’s for commercial work, corporate work, television or cinema production, or esports broadcast.”

What are the benefits of virtual studio technology in TV production?

Crews have been capturing talent against projected backdrops ever since the silent era. Projected backdrops were an impressive in-camera effect. But they also had their limitations: the camera couldn’t move too much, for example, or the backdrop would look flat.

Green screens provided a solution to this problem – but at first, these required extensive compositing work in order to merge real foreground elements to backdrops once the shoot was complete. This could be both time-consuming and expensive.

With today’s virtual studios, all this has changed. Cameras can now be tracked, and gaming solutions like Unreal Engine are powerful enough to render huge landscapes in real time. That means multiple cameras can move across massive 3D scenes in 4K, without having the backdrop look flat or requiring any post production.

Images courtesy of Virtual Line Studios.

There are many other benefits to using virtual studios, including:

• Minimizing costs. If you want to change your shooting location to Times Square, you can do so without any travel costs, VFX costs, or additional permissions. Just use your virtual studio.
• Less risk. Need to reshoot additional dialogue for a winter scene? No problem. You can easily recreate the exact same lighting and weather conditions in a few clicks.
• Full control. With a computer-generated environment, you can easily change time of day or even adjust mountains to suit your creative vision. If you need a sunset to last hours, that’s not going to be a problem.
• Final pixels can be captured on set. You’ll see your final result while shooting on set. That means you can take on projects with shorter deadlines, delivering work without the time-consuming and costly process of changing out physical sets.
• Remote working friendly. If you want someone from across the world to appear on a graphic overlay, virtual studios will make it happen. You can also shoot any world from one location.

What setup do you need to build a virtual studio set? A breakdown.

According to Joe Jett, CEO at Jett Sets and Stefan Popescu, Partner at Safe Frame, there are multiple essential technologies that make up a virtual studio set. These include:

• Green screens and/or LED panels. These can be straight or curved, and can be as large or small as needed depending on the production.
  “We emptied our 500 square-meter warehouse in Bucharest, and built our studio from scratch within just four months,” reveals Stefan.
•  A real-world camera. For Safe Frame, this is Sony FX9 with an 18mm lens on a crane. Meanwhile, the Jett Sets team primarily rigs their F55 camera with a Stanton Jimmy Jib Pro. “We look for cameras with large sensors because the sensors typically have good resolution, and also do very well in low light, so if I’m wearing a darker sweater it will pick up the detail,” says Joe. “A hybrid camera also does well because it can switch to a broadcast look or a cinema look with a LUT. That means you can create a specific look for your project or copy a look that was created earlier.”
• Camera tracking systems. These will be placed on top of the real-world camera and will track it as it moves, translating those movements in 3D space. Jett Sets uses a Mo-Sys StarTracker, while Safe Frame uses VioTrack R from Trackmen.
• Real-time content and compositing software. Safe Frame and Jett Sets both use Zero Density’s Reality Engine, a real-time node-based broadcast compositing system that’s directly connected to Unreal Engine for graphics.
• Design.“There are online libraries [like the Unreal Engine Marketplace] where you can buy assets if a client likes them. We still have to check scales and textures and so on but we can use online libraries as a base,” says Stefan. “We also design from scratch. Typically we advise clients to assign someone as a creative director on the project who will meet with our creative director and they flesh out their ideas and reference images. And we start the modelling process from there into Unreal Engine and Reality.”
• Lighting. Safe Frame uses an Arri Orbiter, Arri SkyPanels, and DMG Lumière Light Kits amongst others.

Virtual Studio of Safe Frame.

How is virtual studio technology helping broadcast creatives do more?

Music videos
In 2020, as the world was forced to reinvent how we can come together for live events, Safe Frame was commissioned for an R&D project. The goal was to add audience members to a recorded performance for the artist Jazzy Jo.

First, the Safe Frame team shot Jazzy Jo and her band against a green screen. Zero Density’s software was then used to composite them into a computer-generated courtyard, all in real time. Next, the team set up a fixed camera and shot two friends dancing in the studio. Using Reality, these friends were added to a 3D balcony overlooking the courtyard scene, where they could dance live to the concert as it took place.

The Safe Frame team also added additional audience members to the scene by finding stock footage of dancers on Shutterstock, and comping them to the balcony as 2D billboards. Ultimately, the team was able to add all real and stock audience members to the set, with very little post work to follow — a workflow that could be replicated for a person on the other side of the world.

Episodics for Netflix
The Jett Sets team is also well-versed in using virtual studio technology to create projects with impressive visuals. They were recently commissioned to help produce a couple of episodes for Netflix’s Bookmarks, a live-action collection of twelve episodes featuring prominent, black celebrities and artists reading children’s books that highlight the black experience.

For Joe, the most challenging aspect of the project, which needed to be delivered in 4K, was the creation of CG books with animated pages that would appear above the celebrity reader, and turn live as the book was read during the show. To do this, the Jett Sets team used Blueprints and Reality to trigger animations like a page turn and closing the book. Each could be seen and controlled live on set as the show was being recorded.

Virtual events
Jett Sets also recently used their virtual studio to help clients create impressive remote corporate events: including a virtual book launch led by creative director Quinn Sterling. Written by a world-renowned knitwear designer, the book would have normally been launched with a live fashion show. Due to the pandemic, however, this was impossible.

Instead, Quinn approached the Jett Sets team, who used their studio to take the viewer through a virtual fashion show that displayed the creative process of designing each sweater, scarf, and cardigan from sketch to finish. Each model was shown standing in a mindscape of the designer, with the book in a frame in the background. In just eight hours, the team was able to capture all 31 outfits in the book using a Sony F55 with a Mo-Sys StarTracker mounted on it. The tracker used LED lights and reflective markers, which were placed along the ceiling of the studio, to calculate the camera’s position in 3D space as well as the lens zoom and focus data. This was fed into Zero Density’s Reality Engine to create the background live in real time, and cue in the various animations. The team was able to see the final image and wrap up on set, without spending weeks keying and compositing.

Talent shows
When Vodafone approached the Safe Frame team to launch a three-month virtual talent show which would be broadcast on Instagram, they were in safe hands. This time, Safe Frame didn’t just design the virtual studio set, but also wrote the script to go with it.

They began building by vibrant virtual stages, including an award ceremony stage and individual performance stages each uniquely customized to suit each finalist. In the final broadcast alone, the Safe Frame team would switch these stages out six times, with just minutes to do so while a prerecorded clip played.

Equally important to having stunning visuals was for the production to launch perfectly. The demand on the equipment was high, throughout the work and especially in the finale, which consisted of a two-hour program following a long day of rehearsals. For many setups, this would push the tech past its limit. But Safe Frame’s virtual studio powered by Reality Engine brought the power and dependability needed for a flawless execution.

Want to learn more about Jett Sets and Safe Frame’s virtual studios? Watch the full webinars here and here!

About Zero Density
Zero Density is a world leader in virtual studio, augmented reality and real-time graphics technologies for the broadcast, live events and esports industries. From the Olympics to Louis Vuitton virtual fashion shows, Zero Density’s Unreal Engine-native platform, Reality Engine — which includes a real-time broadcast compositing system and its proprietary keying technology, Reality Keyer — has been used by some of the biggest companies in the world. Clients include: The Weather Channel, RTL, Fox Sports and Warner Media.