How to Upgrade from MySQL 5.7 to MySQL 8: AWS RDS and W&B

With official support of MySQL 5.7 on it's way out, here's how to upgrade to version 8
Created on January 24|Last edited on January 24
Comment
﻿
Introduction This article walks through the process of upgrading your MySQL database from version 5.7 to version 8 and take advantage of some of the new features and enhancements that come with latest version. While most of the steps can be generally applied for any MySQL 5.7 to MySQL 8 upgrade, this article specifically talks about the upgrade in context of the W&B application.
PurposeWhile Weights & Biases offers few flavors of deployment types, W&B-managed offerings (both SaaS and Dedicated Cloud) are the most popular. This is in large part because deployments remove the operational complexity and maintenance overhead for customers, allowing them to focus on business use cases. 
Still, some of our customers opt for a self-managed deployment. This is most common when they have private infrastructure or a strictly governed air-gapped setup. 
Self-managed deployments require a customer to deploy, maintain and manage few important sub-systems that power our application. These sub-systems include:
W&B Server running on Kubernetes
MySQL database
S3 compatible object storage 
Detailed documentation on how to deploy and configure these components are listed in our official docs﻿
💡
One of the critical aspects in running a fully self-managed deployment is the reliability and performance of the database. Since official support for MySQL 5.7 was dropped in October 2023, W&B strongly recommends upgrading to the latest version of MySQL 8 as soon as possible. 
Let's walk through how to upgrade and what features you'll unlock after doing so. 
A Quick Suggestion Before We Jump InPlan ahead and schedule a downtime window. Make sure to communicate the downtime to the users ahead of time. Database upgrades will result in some downtime for the W&B service so this step is vital to prepare users for any related service disruption.
Upgrade: Code and ExecutionFirst, take a snapshot of your existing W&B RDS database instance. The snapshot might take anywhere between few minutes to a few hours depending on the size of the database. 
 NB: The W&B client itself is fault tolerant, so when the service is down, the client will automatically keep retrying until it’s able to re-establish the connection with the W&B server. Once the server is back up the client will continue as normal without having to stop and restart any jobs that are in progress. Although the client is fault tolerant, it is  limited by the total amount of disk space available, therefore it’s critical to schedule a downtime window and communicate to users ahead of time.
💡
Upgrading from the W&B AWS Terraform ModuleW&B strongly recommends using the official AWS terraform module to deploy the required infrastructure and perform the upgrades to avoid any state drift. You can visit the repo here. 
First, simply update the database_engine_version terraform variable from 5.7 to 8.0.mysql_aurora.3.03.0.
The specific MySQL 8.0 version, 8.0.mysql_aurora.3.03.0 used here was the latest at the time of publishing this blog. Please replace this with the latest version at the time you’re performing the upgrade.
💡
Next, run terraform plan. You’ll notice a plan that might look similar to the following:
~ update in-place
+/- create replacement and then destroy
﻿
Terraform will perform the following actions:
﻿
  # module.wandb_infra.module.database.aws_db_parameter_group.default must be replaced
+/- resource "aws_db_parameter_group" "default" {
      ####
    }
﻿
  # module.wandb_infra.module.database.aws_rds_cluster_parameter_group.default must be replaced
+/- resource "aws_rds_cluster_parameter_group" "default" {
      ####
    }
﻿
  # module.wandb_infra.module.database.module.aurora.aws_rds_cluster.this[0] will be updated in-place
  ~ resource "aws_rds_cluster" "this" {
      ####
    }
﻿
  # module.wandb_infra.module.database.module.aurora.aws_rds_cluster_instance.this["1"] will be updated in-place
  ~ resource "aws_rds_cluster_instance" "this" {
      ####
    }
﻿
Plan: 2 to add, 2 to change, 2 to destroy.
Once applied, this should automatically create the parameter groups and modify the database engine version to the specific version of MySQL 8.
Upgrading from the AWS ConsoleHere, you’ll need to create cluster parameter groups and database parameter groups for 8.0 version in the AWS console. For detailed information on creating parameter groups, please refer to the official AWS documentation. If you’re using the official W&B AWS terraform module, these parameter groups are automatically created.
Once the parameter groups are created: 
Select the database cluster associated with the W&B instance (you can only modify the database engine version at the cluster level. The cluster will then modify the versions for all the instances that are part of the cluster)
Click Modify and change the engine version to 8.0
﻿
Select the right cluster parameter group in the "Additional configuration" section:
﻿
Click Continue and select Apply Immediately or Apply during maintenance window if you have one setup.
You can track the cluster modification events under Logs & Events > Recent events. It provides a timeline of events happening during the upgrade process and might look like this:
﻿
Last, please note that even the smallest database may take up to 10 minutes to finish the full migration. If your database is larger in size the process may take longer. Plan the downtime window accordingly!
SummaryFollowing the above process should help you efficiently upgrade the database version from MySQL 5.7 to the latest MySQL 8.0 version offered by AWS RDS. The database upgrade itself is transparent to the application layer based on the MySQL 8.0.mysql_aurora.3.03.0 version we've tested. Once the migration is complete, the downtime window can be closed and users can resume their ML training runs.
If you have any questions or need further help, please contact support@wandb.com.
﻿
Add a comment
Tags: Articles, W&B Features
Iterate on AI agents and models faster. Try Weights & Biases today.