In less than 2 years, Disney’s Encanto has become a cultural phenomenon.
As the first Disney film to reach 200 million streaming hours on Disney+, the first Disney song of the 21st century to top the US Billboard 100, a chart-topping album, and an Academy Award for Best Animated Feature, Encanto‘s success was unprecedented. This success was also in large part due to the hit songs written by Lin-Manuel Miranda and Germaine Franco, including “We Don’t Talk About Bruno”, “The Family Madrigal”, “Surface Pressure” and Oscar-nominated original song “Dos Oruguitas”.
With the runaway success of Encanto, taking the hit songs to the stage was inevitable. And with so much Broadway caliber talent in the original cast including Miranda, Stephanie Beatriz and Olga Merediz, transitioning their performances to the stage was seamless and destined for success.
Due to the film being produced and released during COVID, Encanto at the Hollywood Bowl was the first time that the cast were able to sing together in person. The cast were also never able to attend any press junkets or conventional movie premieres due to the pandemic.
With this in mind, Encanto at the Hollywood Bowl was positioned to be the homecoming and premiere event that the film always deserved.
AOIN Involvement All of it Now was introduced to the Fulwell 73 team via North House, and worked in close collaboration with the Fulwell 73 and NorthHouse teams to establish a design philosophy and language for the augmented reality (AR) elements. It was quickly determined that the AR elements would best serve as extensions of the projection content, using the vertical space above the Bowl as the creative canvas.
The North House team shared assets used in the projection content, enabling creative cohesion between projection and AR elements. Both teams used Unreal Engine for the creative content, which became a shared collaborative platform across both teams.The 3D meshes of the venue were also used for dynamic occlusion of the AR elements so that AR objects could be placed both in front and behind the bowl and other set elements.
All of it Now worked closely with the Stype team to track and calibrate 4 cameras for use in the live broadcast as well as to record camera tracking data for use in rendering out AR plates, which the post-production team overlaid onto the UHD camera records.
Technical Challenges The AR content pipeline needed to serve two use cases: 1. Live broadcast AR visible to the audience. 2. A post-production pipeline for the streaming audience on Disney+.
For the live AR component, content needed to be optimized for real-time playback in Unreal 5.03 to then be re-rendered in UHD for the streaming version on Disney+.
With both real-time and post-production workflows requiring seamless and parallel pipelines, Unreal Engine was the natural choice for the project.
The first step in the creative process was accurately representing the venue so that all creative stakeholders could accurately visualize the venue and creative elements in context. All of it Now took a multidisciplinary approach to creating the key Hollywood Bowl geometry assets, using a combination of photogrammetry, LIDAR scans, and conventional modeling. The model was then used in multiple stages of previsualization, with the team working in close collaboration with camera director Chris Howe, and Director of Photography Cameron Barnett.
Together, the team performed camera and lens studies within Unreal Engine 5, selecting the best cameras and lenses to capture the AR moments. The team used the 3D stage meshes and camera plot to model lensing options, which enabled swapping out lenses in live previz sessions with the show’s director. Cameras were selected for AR including 2 moving cranes and 2 tripod cameras, giving the director a variety of perspectives throughout the venue, and enabling creative continuity across multiple camera angles.
Upon identifying the best camera positions for AR, the AOIN team began the creative previz process, simulating camera moves that were informed by the range of movement of the AR enabled cameras to find the best camera angles for each moment.
With a quick timeline for post-production, using the camera tracking recordings from the Stype cameras was essential for delivering the 4k HDR renders for the Disney+ platform.
AOIN delivered over 36 AR 4k plates in 1 week using a custom render pipeline utilizing Unreal 5, Stype Movie Render Queue, Adobe Premiere, Nuke Studio, and Deadline.
The Result By overcoming these challenges through technical innovation, AOIN delivered a celebration capturing the essence of Encanto by blending projection content and AR content into a unified visual experience blurring the lines between reality and fantasy.
