NVIDIA Unveils AI Reference Apps for Real-Time Media Effects on Holoscan

NVIDIA has announced the release of AI reference applications designed to enhance real-time media workflows, according to a recent blog post by Guillaume Polaillon on the company’s developer website. These applications, available on NVIDIA Holoscan for Media, address the challenges posed by high-bitrate, uncompressed media streams, which are often hindered by network latency and bandwidth constraints when processed with cloud-hosted AI models.

AI Reference Applications

The new AI reference applications serve as a robust foundation for developers aiming to create real-time AI solutions for live media workflows. These applications can efficiently handle uncompressed ST 2110 streams, enabling real-time media effects with minimal latency.

One notable application is the AI virtual camera, which utilizes PyTorch and NVIDIA DeepStream SDK. This tool creates virtual cameras for each presenter in a video by detecting and tracking individuals in a high-resolution ST 2110 input stream. The result is multiple cropped virtual camera outputs focused on each detected individual, allowing operators to generate dynamic production shots using a single static camera setup.

Another application leverages real-time automatic speech recognition (ASR) on ST 2110-30 audio sources using NVIDIA Riva Parakeet ASR NIM. This tool provides a web interface that displays live transcription and allows users to search for specific words. This setup offers developers a base to refine and customize further.

Getting Started

For those interested in utilizing these AI reference applications, NVIDIA recommends having an AI workstation equipped with an NVIDIA RTX Pro GPU and a ConnectX network interface card, or a certified multi-GPU system. Additionally, developers should set up an NVIDIA Holoscan for Media environment using either a local developer setup with Kubernetes or a platform reference deployment guide with a jump node.

Further installation guidance and resources are available on the Holoscan for Media page.

Additional Updates

The Holoscan for Media 25.4 release also includes enhancements in monitoring for both production and local developer environments. This update features improved automation for single-node OpenShift installations and compact three-node clusters, supporting more networking variants and simplifying the automation of local developer setups.

The release also supports automated installation of reference applications, including the Helm dashboard and NMOS registry.

Conclusion

Since its launch, Holoscan for Media has facilitated container orchestration for multi-vendor live production. The latest AI reference applications in the 25.4 release deliver on the promise of real-time AI for live media on premises, offering developers new tools to enhance their production capabilities.

Image source: Shutterstock