Devsisters Levels Up Game Development with W&B
"W&B helped us a lot in model development and model updates,” said Yunyol. “Without the visualization of our metrics, it’s very hard to know where the problems are and see what we should change."
Yunyol Shin
Applied Data Scientist
Keeping Players Engaged
When it comes to mobile games, you either become a user’s break time essential or lose them in the first few weeks of download. With almost 700,000 mobile games available in the app stores, it’s not surprising people feel compelled to explore the latest and greatest.
With a franchise like the Cookie Run IP, loved by more than 200 million users worldwide, global entertainment and gaming company, Devsisters, knows a thing or two about the makings of a loyal fanbase.
Harnessing Reinforcement Learning
Mobile games are often known for their addictive gameplay. And to keep players engaged for hours on end means keeping the game balanced and bug-free—after all, it’s no fun when a single strategy beats all the rest or encountering glitches along the way.
Typically, creating a positive player experience requires thousands of play-testing sessions from test users to tune the balance of the game and ensure it’s free of critical and noticeable issues. This process is not only time-consuming but also imperfect—the more complex the game, the easier it is for subtle flaws to slip through the cracks.
At Devsisters, the Data Science team has turned to reinforcement learning (RL) to build a robust game-testing system, and this internal solution is currently used to optimize their upcoming game, Cookie Run: Witch’s Castle. Set in the fan-favorite Cookie Run universe, players can experience a never-before-seen adventure and enjoy the “tap-to-blast” puzzle action, decorate the Cookies’ rooms, and much more.
At its core, Cookie Run: Witch’s Castle, is a puzzle adventure game with many stages for players to gradually unlock. The success rate of these stages has a significant impact on player engagement and retention. For instance, if the tutorial stages designed to be easily solvable would require numerous attempts to complete, players could lose interest and abandon the game.
Using the RL system, the team relies on an intelligent agent to play the game and adjusts the stages accordingly depending on how well the agent performs. “We think of our system like a CAD tool but for game developers,” said Yunyol Shin, Applied Data Scientist at Devsisters.
And the system’s capabilities don’t stop there—it can also catch bugs, especially the ones that are hard to find during build updates. Some of these bugs occur in specific states that can only be surfaced by playing thousands or millions of rounds. Some of the game developers at Devsisters say it has shortened a development cycle from 2-3 days to 3-4 hours.
With the rapid iteration of game development, the team relies on a model coined Nimbus that acts on the principle of training fast and playing well to optimize the stages in a two-week cycle. The system finishes training for 50 stages in a single day with an elite-level performance from the agent.
Here is an in-company experiment comparing the success rates of human players vs. the agent:
Normally, play testing games and identifying imbalances at the level Nimbus provides could take months. With the RL system in place, that time has reduced significantly—launching games to the market faster and better. And for a company like Devsisters situated in a hyper-competitive industry, there is a definite need for speed.
Deriving Insights in Real-Time
When it comes to game development, it is a highly complex process that requires iteration and experimentation. In developing their new game, the team had to retrain the RL agent for every update which happened roughly in a two week cadence. This means the distribution of the parameters internal to the model goes through constant change and having the ability to observe those variations and act upon them becomes key. “With W&B we can observe the parameter distributions changing continuously and optimize them to achieve maximum performance,” said Yunyol.
Additionally, W&B’s powerful visualizations became a critical tool in helping the team debug model performance. With the fast feedback loop and the ability to see exactly where the problem areas were, the team could address the bug immediately and work on getting a fix. “W&B helped us a lot in model development and model updates,” said Yunyol. “Without the visualization of our metrics, it’s very hard to know where the problems are and see what we should change.”
Along with the human resources it takes to build a game, costly hardware also comes hand in hand. W&B provides a centralized place for the team to track system utilization giving them insight into the training bottlenecks and possible ways to uplift it. By being able to see live metrics like GPU utilization in real time, the team can quickly make the appropriate adjustments to avoid wasting expensive resources.
Empowering Game Developers
As the gaming industry grows more vast and complex, helping development teams build better games efficiently will be key to staying ahead of the competition. While Devsisters has seen global success with their flagship games, Cookie Run: Kingdom and Cookie Run: Ovenbreak, the team hasn’t stopped there but instead is finding new ways to accelerate game production and elevate the player experience.
To challenge the traditional game development workflow, Devsisters relies on W&B to bring their models to the best state for deployment. Through powerful, extensible visualizations, Devsisters can debug easily, build iteratively and deploy with confidence. Innovation and experimentation are critical in the gaming industry, and W&B is here to support Devsisters to build the future of game development.
For current and soon-to-be Cookie Run fans, keep an eye out for updates on the new tasty addition, Cookie Run: Witch’s Castle. The timer on the release date is ticking away, and soon we can all go on another sugary adventure with a spooky twist.
