For Coachella 2022, All of it Now worked alongside Epic Games, &Pull, Six Degrees of Freedom, and Flume’s creative team, led by TRASK HOUSE’s John McGuire, to develop the AR system for the Main Stage livestream. Following the festival’s two-year hiatus, the project extended the visual experience beyond the physical stage, creating augmented content that existed specifically within the broadcast.
For Flume’s performance, All of it Now handled the final integration of the AR assets into the live production, along with technical development around camera placement, lens selection, system engineering, and playback. The system operated across three tracked broadcast cameras, allowing Unreal content to remain aligned with the physical stage from multiple perspectives.
The AR pipeline was built in Unreal Engine 4.27.2 using stYpe tracking and the StypeLand plugin. Custom 3D holdout tools were developed to account for the physical stage within the Unreal environment, allowing virtual objects to pass behind scenic elements, interact with the stage, and maintain accurate depth relationships within the camera feed. A custom timecode system was also developed to trigger and change Unreal sequences during the live performance.
Lighting data was integrated through DMX, allowing virtual lighting to follow the same cues as the physical production. This helped connect the AR elements to the lighting environment on stage, rather than treating the virtual content as a separate layer within the broadcast.
The project also included a series of AR interstitials designed specifically for the livestream. These pieces pushed the holdout and compositing system further, with reflective and interactive virtual objects requiring accurate relationships between the Unreal environment, the physical stage, and the tracked camera.
With only a few weeks of development followed by several days of on-site integration, the system had to move quickly from creative development into a live broadcast environment. The final pipeline connected Unreal Engine, camera tracking, custom real-time tools, lighting, and broadcast integration to extend the Coachella Main Stage into an AR environment for the livestream audience.
