Bring your own bucket with CoreWeave AI Object Storage
We’re introducing a new integration between CoreWeave AI Object Storage and Weights & Biases. Here's what you need to know
Created on June 27|Last edited on June 27
Comment
We’ve added CoreWeave AI Object Storage as a storage provider for files and artifacts logged into Weights & Biases, enabling secure, ultra-fast data loading and model checkpointing for large-scale training and fine-tuning jobs.
Fig 1: Configuring CoreWeave AI Object Storage in Weights & Biases
When training and fine-tuning AI models at scale, loading datasets and saving intermediate checkpoints must be as fast as possible. Otherwise, you risk wasting valuable GPU resources and AI engineer time. Accelerating data access is also crucial for improving experiment velocity, as delays across multiple runs can accumulate into weeks or even months.
That’s why we’re introducing a new integration between CoreWeave AI Object Storage and Weights & Biases. You can now use the Weights & Biases’s bring-your-own-bucket (BYOB) feature with CoreWeave AI Object Storage, the AI industry’s first managed object storage service designed specifically for maximizing throughput performance in AI workloads. This integration enables you to securely store artifacts and other sensitive data directly in your existing CoreWeave cloud.
Bring your own bucket (BYOB) for sensitive workloads
With BYOB for dedicated single-tenant and SaaS multi-tenant deployments of W&B Models, data stored in your bucket isn’t copied to Weights & Biases managed infrastructure, enabling you to comply with stringent standards for highly sensitive data. When using BYOB, experiment files, artifacts, media (images, videos, audio), run files, and history metrics and system events exported in Parquet format remain securely in your bucket, providing essential data isolation and security.
Previously, BYOB supported Amazon S3, Microsoft Azure, and Google Cloud Platform buckets—and today, we’ve expanded your options by adding CoreWeave AI Object Storage, the latest in optimized storage for AI workloads. Learn more about Weights & Biases BYOB.
What’s CoreWeave AI Object Storage and why does it matter?
CoreWeave AI Object Storage is an exabyte-scale, Amazon S3-compatible storage platform designed specifically for GPU-intensive AI model training and inference. It scales to hundreds of thousands of GPUs, enabling organizations to train large language models (LLMs) and other foundation models more rapidly. This accelerated performance reduces training times and overall costs, speeding up iteration and innovation cycles.
CoreWeave AI Object Storage can be used with CoreWeave’s Local Object Transport Accelerator (LOTA™), which accelerates data access by caching and pre-staging objects directly onto GPU nodes’ local disks.
Fig 2: Local Object Transfer Accelerator
Here are the key benefits of CoreWeave AI Object Storage:
- High performance results: See a significant per-GPU performance boost with up to 2 GB/s/GPU, even across hundreds of thousands of GPUs.
- Greater capacity: Scale to trillions of objects and exabytes of data.
- High reliability: Get the benefits of 99.9% uptime and 11 9’s of durability, so your team can get models to market ultra fast.
- Compliance and support: CoreWeave AI Object Storage utilizes a standard S3 interface and supports open-source S3 SDKs.
- Easy integration: Tight integration with CoreWeave Kubernetes Service (CKS) and SSO/SAML.
- Security built in: Authentication and policies with roles-based access. Encryption at rest and in transit.
Bringing CoreWeave AI Object Storage to Weights & Biases
You can now configure CoreWeave AI Object Storage buckets as a supported storage format for W&B Models dedicated cloud deployments, with support for SaaS multi-tenant cloud deployments coming soon.
With CoreWeave AI Object Storage as your object storage bucket in W&B Models, you benefit from high-performance storage with minimal code changes. Once configured for your team or instance, your artifacts and files are automatically stored in CoreWeave clusters. Additionally, you can leverage LOTA with reference artifacts to achieve maximum throughput when downloading datasets or model checkpoints.
Learn more about BYOB in the data security section of our documentation. Contact the Weights & Biases support team at support@wandb.com for assistance setting up CoreWeave AI Object Storage as BYOB provider.
Add a comment
Tags: Articles
Iterate on AI agents and models faster. Try Weights & Biases today.