Leveraging AI for Visual FX at MARZ
"Once people started seeing the value of W&B, it kind of just exploded, and everyone on the ML team now builds everything on it in the company"
Thomas Davies
Research Team Lead
An Emmy-Nominated Startup
WandaVision, Watchmen, and Wednesday—what do these films have in common? Stunning visual effects (VFX).
Coincidently, they were all also produced by the same VFX company: MARZ, short for Monsters Aliens Robots Zombies. It’s no surprise that to present such movie magic takes a lot of resources and money, or at least that used to be the case. MARZ is an AI-enabled VFX studio disrupting the industry with a reputation for leveraging technology to deliver a best-in-class product with record timelines. In the age of bingeable, streaming television, MARZ is opening the door for infusing machine learning (ML) and VFX—saving time, effort, and cost.
Driving Efficiency in Entertainment
When VFX comes to mind, one naturally thinks of creating creatures that don’t exist, like monsters, aliens, robots, or even zombies. The other application that hides from the spotlight (and that audiences rarely notice) is the mundane but necessary work to ensure continuity.
Continuity in filmmaking is the principle of maintaining consistency in all the details from shot to shot and from scene to scene. Each shot should feel like it flows seamlessly from the previous shot, reinforcing a sense of realism. But to stay consistent and maintain fluidness and continuity is tough, to say the least. Some production crews say they spent at least 80% of their effects budget fixing visible wig lines
At MARZ, the goal of their ML team is to build tools for creatives to speed up traditionally manual and cumbersome workflows, allowing them to focus on the art. Their first ML product, Vanity AI, is the first end-to-end Hollywood-ready ML tool built for aging, de-aging, prosthetics, and digital makeup. The technology is 300 times faster than traditional VFX pipelines, significantly more cost-effective, and has no capacity constraints.
Take, for example, under-eye bags. Historically, VFX studios rely on a manual process where artists have to trace out the bag, apply some additional effects to the region, and rinse and repeat that for every single frame, every single shot, across the entire film. With Vanity AI, artists can automatically apply eye bag removal, and it’s as simple as moving dials and selecting a result that matches the desired edit.
Before and After Edits Using Vanity AI
Evaluating Models with W&B
While ML is heavily used at MARZ now, that wasn’t always the case. MARZ existed for five years before having any ML teams in place. This poses an interesting advantage for MARZ. It means the company has plenty of past work they can use as training data to build their models.
Good quality training data, access to scalable GPU compute infrastructure, and the right developer tools are critical parts of the model development process. MARZ leverages both NVIDIA GPUs and Amazon Web Services for access to scalable data storage and high performance compute resources. To ensure the best models get pushed to production, MARZ relies on purpose built AI developer tools such as W&B Tables.
For each shot, MARZ generates a table that helps them track every little detail, and more importantly, understand high-level patterns. The table below shows the team the pixel error on a per-frame basis for their various experiments.
On top of that, a summary table is created to see the overall errors for all of the shots.
Using Tables, the team captured vital information allowing them to easily see which particular frames they struggled with—enabling them to move quickly and come up with the next steps. “W&B gave us the tools to build out testing processes specific to our data,” said Thomas Davies, Research Team Lead at MARZ.
To include additional context about their tables and document other key findings, MARZ leverages W&B Reports. Many of MARZ’s reports are a shot-by-shot analysis exploring how their different experiments performed. The team uses reports regularly, so everyone has visibility into what’s going on, keeping the entire team on the same page.
While Thomas was the initial champion of W&B, today, everyone at MARZ runs their experiments in W&B. “Once people started seeing the value of W&B, it kind of just exploded, and everyone on the ML team now builds everything on it in the company,” said Thomas.
Allowing Creativity to Thrive
What started as founding its own AI department to deal with the shortage of VFX capacity has transformed into one of the fastest-growing VFX studios in the industry. The tools MARZ is building remove the redundant steps that artists often face in their workflow and allow them to focus on what matters most: their art.
To ensure they are developing the best AI for VFX solutions, MARZ relies on W&B to analyze and understand datasets, measure and debug model performance, and share key insights across the team. MARZ is disrupting the entertainment industry in a way that puts artists’ needs first, and W&B is thrilled to play a small part in their endeavor.
To learn more about Vanity AI, check out MARZ’s demo video here!
