TensorStore: Google AI's Solution To Working With Massive Multi-Dimensional Data
TensorStore is Google's solution to managing massive multi-dimensional data arrays efficiently and safely, with build in storage integrations.
Created on September 26|Last edited on September 27
Comment
For machine learning engineers and data scientists, working with massive multi-dimensional data arrays is pretty standard - especially on large-scale projects. Because it takes time and storage to handle large amounts of data, working with it efficiently is key to a streamlined workflow.
The researchers at Google AI aren't exempt from this fact, so they put together TensorStore to manage their data, and recently formally introduced it.
TensorStore is a library (built for both C++ and Python) that aims to make working with tensor data safer and more efficient with native support for existing storage systems, memory-optimized data interaction, multi- and remote-access operations, and more.
Researchers at Google have already used TensorStore in developing their own machine learning models, such as PaLM and its hundreds of billions of parameters, for managing model parameters during training. They have also used it for storing massive datasets, such as the full 3D mapping of a fly's brain, or 1 cubic millimeter of human brain tissue represented by 1.4 petabytes of data.
Get using TensorStore
You can get started with TensorStore by going to its documentation website (check here for installation and here for a tutorial). You can also take a peek and its GitHub repository if you're into that.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.