* This interview originally published on FEED
There is an explosion of virtual set technologies. Once reserved for top-tier broadcasters, video magic is now becoming available to anyone.
Virtual sets are now a common sight on our screens, replacing the plywood and fabric of studios past and placing presenters, pundits and news anchors in real-time rendered, shiny-floored virtual spaces, complete with augmented reality pop-ups and CG characters.
Today, a virtual studio means more than just a stage with a green screen, says Ulaş Kaçmaz, vice-president of sales and marketing for Zero Density, which specialises in real-time graphics and VFX. “Augmented reality elements, hybrid studio operations – part green screen, part physical – portal windows, set extensions and so on also mean virtual studio production. Virtual studio use is surely expanding, although without a hint of physical sets becoming obsolete at any time in the future. There are immense possibilities in other areas for this technology to enhance visual storytelling. Creating a fantasy world from scratch and adding texture and spirit with invisible real-time VFX.”
Kaçmaz points out the potential for virtual set-based documentaries and historical remakes. “They already have such strong stories,” he says. “Imagine the captivating picture that could be curated by employing this tech. Also, you’d be in complete control of your scene during the shooting or previs stage instead of fixing all up in the post and settling for the inevitable glitches.”
“I think content creators like the versatility of having a physical set and being able to bring virtual elements into it to add some extra creativity to their programming. It’s definitely becoming more widely used in traditional programming and the sports and esports industries have been right at the front edge of the technology. Ross recently worked with esports company ESL and Spidercam to help deliver augmented reality content during the ESL One event in Cologne and I think we’re going to see this kind of content becoming more prevalent.”
“The market will definitely continue to increase exponentially,” agrees Andy Hook, technical solutions director at broadcast gear provider, White Light. “Right now, we are seeing a shift in a virtual studio being more acceptable for use in corporate, education and live streaming environments. We’re also seeing more hybrid mixed reality studio sets, combining real sets with virtual extensions to make them feel bigger or create elements that would be physically or financially difficult to construct.”
Virtual sets technology works at its best when it complements the story rather than being gimmicky
“Storytelling is the key point,” agrees Ross Video’s Russell. “Virtual sets technology works at its best when it complements the story rather than being gimmicky, and I think the technology continues to develop in a way that makes the images we produce more realistic, natural and interesting to the viewer.”
An essential part of virtual set solutions is accurate camera tracking. Systems from the likes of Mo-Sys, Stype and Ncam allow live tracking and precise placement of virtual and physical elements. Virtual set software converts the data coming from the tracking system and converts it to virtual camera data for the rendering engine.
“Ncam’s approach is to provide a complete virtual set solution, meaning that not only do we provide the most flexible and robust camera tracking technology available, but we also supply highly accurate lens distortion data, which is crucial to ensuring that the real world matches up seamlessly with the virtual world,” says Nic Hatch. “Ncam’s recently released AR Suite powered by version 4.23 of the Unreal Engine is able to take all the Ncam data and produce a final keyed and composited image. All these elements have been designed from the ground up internally at Ncam.”
Until now, it has been the privilege of a select group of broadcast organisations to use high-end virtual studio technology
This integration of the Unreal Engine (UE) from Epic Games as the 3D renderer in many proprietary virtual set solutions has pushed the wow factor for a couple of years now, but tools built around this photorealistic real-time game engine continue to impress and be refined. An example is the Reality Engine composting software from Turkey-based Zero Density.
“Reality Engine’s unique architecture allows Hollywood-level visual effects in live production and the node-based compositing and keying tools enable the most photorealistic output,” claims Kaçmaz. “Zero Density not only provides the main system, but also the necessary automation, monitoring and controlling interfaces. The system is designed to run on multiple Unreal Engine instances that are all controlled and managed as one.”
“Our Reality Keyer product works with a clean plate and combines this with the system’s tracking functionality to produce a mesh representation of any studio green screen cyclorama,” continues Kaçmaz. “It’s implemented as a shader inside UE4. The system’s use of projection mapping of the clean plate assists the keying and makes the system much more advanced than just a normal chroma keyer. It is the world’s only real-time image-based keyer that works on GPU, preserving real environment details such as contact shadows, transparent objects and sub-pixel details like hair.”
VooSport and BeTV decided to go virtual for their L’Europe des onze programme covering European football. The show was powered by Reality from Zero Density. The Belgian pay TV channel was based in Keywall studios in Charleroi, Belgium.
“Dreamwall designed the set and powered it with seven Reality Engines on seven cameras, including one crane,” says Kaçmaz. “The weekly show has already recorded more than 320 hours live from the virtual studios. They wanted to transform the game nights with seamless commentaries with 3D graphics, interactive elements and visually stunning content.”
VooSport and BeTV use virtual sets for L’Europe des onze, transforming the programme with 3D graphics
Aximmetry is a Hungarian company that had its start in designing video solutions for stage. Its most recent focus has been seamless integration of real and virtual environments and, to this end, the company has created an internal, software-based keyer as part of its virtual studio software.
“The new advanced keyer provides superior results for keying transparent objects, contact shadows and fine details,” says chief operations officer Orsolya Dormon. “It can handle broadcast-quality real time in 4K with GPU-accelerated rendering at a fraction the GPU usage. We have also made a leap in generating realistic shadows. Apart from contact shadows, Aximmetry can cast virtual shadows on the talent and we can combine these seamlessly with the real-life shadows generated by the talent itself.”
A mixed reality environment is the standout feature in White Light’s virtual studio tool, SmartStage. “Our solution omits the need for green screens by instead utilising LED video walls, so the content is really there,” says White Light’s Andy Hook. “This makes the system much more user-friendly. You do not need to be trained in broadcast, you can just walk on set and the interaction, eyelines, reflections, lighting and colour is all real and more natural.”
Next big thing
“Making real-time ray tracing a reality has been the biggest advancement of this year, promising true-to-life scenes,” says Zero Density’s Kaçmaz.
In this case, software has had to catch up with the hardware. “Nvidia announced the RTX cards dedicated to real-time ray tracing last year,” he adds. “Zero Density and Nvidia partnered with Fox Sports for daily presentations inside a real-time ray-traced virtual studio at NAB 2019, which was solid evidence it can be done. At IBC2019, NEP The Netherlands and Zero Density showcased a superior real-time ray-traced virtual studio. Creative and developer teams have not adopted this tech completely yet, so it is not mainstream, but will be starting from 2020.”
* You can read the full story on FEED