A Gentle Intro To Tensors With Examples

In this article, we'll dig into tensors. We'll look at what they are, how to use them, and some of their real-world applications.
Vincent Tu
Created on December 14|Last edited on January 4
Comment
﻿
IntroductionChances are if you've dabbled in machine learning before, you've heard of tensors. But what is a tensor, really? Is it just a giant array of data? In this article, I'll be exploring what a tensor is, use cases of tensors, and examples of them in action.
Here's what we'll be covering:
Table of ContentsIntroductionWhat Are Tensors In Machine Learning?How Do We Determine the Rank/Dimension of a Tensor? How Can We Describe a Tensor's Shape?What if the 2D Arrays Within three_dimensional_tensor had Different Shapes?What About the Tensor's Data Type and Size? Tensors in MathematicsWhat is the History of the Word "Tensor" in Mathematics?What are Tensors Used For?How are Tensors Used in Machine Learning? Why are They Important?Tensors In PythonUsing Weights & Biases With TensorsExamples Of Tensors In ActionConclusionRecommended Reading
﻿
What Are Tensors In Machine Learning?In machine learning, a tensor refers to some multi-dimensional array of data. You can generally think of a matrix as a rank-2 tensor. The only notable difference is that tensors can be of any rank greater than or equal to 0, where the rank is how many dimensions is in that tensor/array.
A tensor has 4 main attributes: 
rank: also called dimension; the number of axes of a tensor
shape: an n-tuple where n is the rank; each value in the tuple is the size of that particular dimension
data type: the type of value/data of the tensor (e.g., float, integer, object, etc)
size: how large an axis is; can also mean all the axes multiplied together
Notice in this context, rank is synonymous with dimension, and array is synonymous with tensor. In some cases, dimension might be a substitute for axis. 
two_dim_tensor = [
	[1, 2, 3],
	[4, 5, 6]
]
The tensor we just described is 1-dimensional or has a rank of 1. Now take the list of lists of numbers defined above two_dim_tensor. This 2D array or tensor would be of rank-2. 
(If any of that is a bit confusing, or you're a particularly visual learner, the figure in the next section should be helpful!) 
How Do We Determine the Rank/Dimension of a Tensor? We can think of the rank as a question: how many indices do we need to select any specific value within the tensor? 
Let's try to find the number 3. For the 1-dimensional tensor, we only need to index once: the 2nd element (counting from 0) is 3. For our 2-dimensional tensor, we need to index twice: we index first into the zeroth list, then find the 2nd element, which is 3.
To better understand the pattern here, refer to Figure 1. Notice that with each new dimension, we are stacking a list of tensors from the previous dimension. A 3D tensor is simply a collection of 2D tensors stacked together, much like how a 6D tensor is a stack of 5D tensors. 
	Figure 1. Tensors of different dimensions. Source: knoldus﻿
﻿﻿﻿﻿﻿
﻿﻿How Can We Describe a Tensor's Shape?We can describe a tensor's shape with a tuple of integers. For our 1D example above one_dim_tensor, it has a shape of (3,). However, there is a slight nuance in working with tensors, especially in popular deep learning frameworks like TensorFlow or PyTorch.
horizontal_tensor = [1, 2, 3] 
vertical_tensor = [ 
	[1],
	[2]
	[3]
]
horizontal_tensor has a shape of (3,), but vertical_tensor has a shape of (3, 1). The extra brackets around each number do matter! Technically, the vertical_tensor has a rank of 2, not 1 like the horizontal_tensor.
We've seen 1D and 2D tensors; below is an example of a 3D tensor. Its shape is (2, 2, 3) because the outermost brackets have two 2D tensors, hence the first 2 in (2, 2, 3). Both 2D tensors within are of shape (2, 3), hence the second 2 and the 3 you see in the shape (2, 2, 3).
three_dimensional_tensor = [
	[
		[1, 2, 3],
		[4, 5, 6]	
	],
	[
		[7, 8, 9],
		[10, 11, 12]
	]
]
What if the 2D Arrays Within three_dimensional_tensor had Different Shapes?That's called a ragged tensor! A ragged tensor is a tensor that has a varying number of elements along any axes. Let's see an example:
ragged_3d_tensor = [
	[
		[1, 2, 3],     # 3 elements.
		[4, 5]         # 2 elements.
	],
	[
		[6, 7, 8, 9],  # 4 elements.
		[10, 11, 12]   # 3 elements. 
	]
]
﻿
# ragged_3d_tensor shape: (2, 2, *)
ragged_3d_tensor would be of shape (2, 2, *) because each column within the 2D tensors is varying, but the number of rows for both 2D tensors are consistent. Keep in mind, for this example, the only varying axis is the last axis. The tensor could have all 3 axes vary. However, the tensor would still be rank-3. 
What About the Tensor's Data Type and Size? There are many different types of data types that work with tensors (you can see an example list of data types on TensorFlow). As you'd likely expect, the data type describes the type of data in the tensor. 
As for size, this can imply a particular axis or the entire size of the tensor. The three_dimensional_tensor has a total size of 2 * 2 * 3 = 12. However, the size of axis 0 is 2 (the first 2 in the shape). The size of axis 1 is 2, and the size of axis 2 is 3. The total size is simply the number of elements in the tensor. 
Tensors in MathematicsThough "tensor" may mean something slightly different in different fields, it's fundamentally the same core definition. Our ML definition of tensors (a multi-dimensional array of data) is fine for our purposes. But for clarity, let's discuss the origin of the word.
What is the History of the Word "Tensor" in Mathematics?Originating from Latin tendere which means "to stretch", "tensor" emerged in the late 19th century. In 1884, Josiah Willard Gibbs, a scientist in mathematics, physics, and chemistry, used this term in his study of strain on the human body. In 1894, Woldemar Voigt used this term in his study of the physical properties of crystals.
Both of these early usages of "tensor" define what mathematicians understand it as today: a tensor is "an algebraic object that describes a multilinear relationship between sets of algebraic objects related to a vector space" (Wikipedia). 
tensor: an algebraic object that describes a multilinear relationship between sets of algebraic objects related to a vector space
💡
This definition is better understood with an example and some context.
An algebraic object is just something in mathematics that can be formally defined. Understanding what a vector space is, a multilinear relationship is a mapping function...
f:V1×V2×...×Vn→Wf: V_1 \times V_2 \times ... \times V_n \rightarrow Wf:V1​×V2​×...×Vn​→W﻿
...where if you keep everything but vi∈Viv_i \in V_ivi​∈Vi​﻿ constant, then f(v1,...,vi,...,vn)f(v_1, ..., v_i, ..., v_n)f(v1​,...,vi​,...,vn​)﻿ is a linear function of viv_ivi​﻿. This mapping function fff﻿ is a tensor!
From its physics and mathematics roots, "tensor" has now taken on a different definition in light of machine learning.﻿﻿
What are Tensors Used For?The usage of tensors spans multiple fields including physics, mathematics, and machine learning. In the mathematics department, simply put, tensors are a standardized format that possess unique transformation properties convenient for a lot of tasks. In machine learning, tensors–in conjunction with better hardware systems–allow for rapid processing of huge amounts of data. 
How are Tensors Used in Machine Learning? Why are They Important?The use of tensors in machine learning and their importance lies in how suitable they are for processing data. 
Machine learning, especially deep learning, benefits greatly from parallelized computation. Coupled with explosive growth in hardware (GPUs and TPUs), it brewed the perfect breeding ground for the crossing of parallel computing and ML. 
The inputs, outputs, and "neurons" (or weights and biases (yes, pun intended)) of deep neural networks (DNNs) are organized into matrices or tensors. Every time the input data is fed into a DNN, a chain of matrix multiplication occurs. Because "neurons" and input data are in tensor format and because we have hardware like GPUs and TPUs that specialize in parallel computing, a huge amount of arithmetic can be done at the same time, speeding up processing by thousands of orders of magnitude. In fact, this spectacular feat backs all of modern deep learning today!
Let's see an example of tensors in ML:
Figure 3. Tensors in A Shallow Dense Neural Network. Source: sausheong﻿
Don't worry if you have no idea what the above image is! They are called Artificial Neural Networks (ANNs) or Neural Networks for short. This concept is the basis for all of deep learning. Loosely inspired by how our brain's neurons are wired, this network has its own neurons and connections. Each circle is a node, and each node is fully connected to all nodes to the right of it. All the gray lines and circles represent the neural network. The input goes in i1,i2i_1, i_2i1​,i2​﻿ and the neural network processes it and outputs o1,o2o_1, o_2o1​,o2​﻿. 
With this context established, notice how convenient it is to pass data into the network. We organize the input into a vector or a rank-1 tensor (i1i2)\begin{pmatrix}i_1\\i_2\end{pmatrix}(i1​i2​​)﻿of shape (2,). Our weights can be organized into a rank-2 tensor (w11w12w21w22w31w32)  \begin{pmatrix}
    w_{11} & w_{12}\\ 
    w_{21} & w_{22}\\
    w_{31} & w_{32}\\
  \end{pmatrix}​w11​w21​w31​​w12​w22​w32​​​﻿of shape (3, 2). Our outputs are organized into another rank-2 tensor (o1o2)\begin{pmatrix}o_1\\o_2\end{pmatrix}(o1​o2​​)﻿ of shape (3,). Instead of computing w11×i1w_{11} \times i_1w11​×i1​﻿ and so on in a sequential fashion, by formatting the weights and inputs into matrices and computing this on processing units like the GPU, we can compute all of this arithmetic at the same time! Now when we train this neural network on data, this entire process can be done much faster compared to computing every single addition or multiplication separately. 
Tensors In PythonLet's get our hands dirty, shall we? For our experiments today, we will be using the follow technologies: Python, NumPy, and TensorFlow. For brevity, since TensorFlow and PyTorch have similar tensor objects, I'll showcase TensorFlow's tensors. Before we begin, let's set up our environment first. 
You can either install a standalone Python copy or install Anaconda (a nice Python distribution that comes with all the necessary data science and ML libraries). 
Install Python:
Documentation: https://www.python.org/downloads/﻿
Article Tutorial: https://realpython.com/installing-python/﻿
Install Anaconda
Documentation: https://docs.anaconda.com/anaconda/install/index.html﻿
Windows: Installing Anaconda, Jupyter Notebook, and Spyder on Windows 10 by ProgrammingKnowledge﻿
Mac/Linux: How to Install Anaconda on MacOS by TechTFQ﻿
In this case, I would recommend installing Anaconda and using a Jupyter Notebook. The IDE used does not matter, but for experimenting and playing with code, I like to use Jupyter Notebooks.
Alternatively, if you want to avoid the installation hassle altogether, you can use Kaggle notebooks (all you need is a Kaggle account) or Google Colab (needs Google account) or Deepnote (just needs a Google account to link to). For the sake of tensors, any of these interactive notebooks are fine (I prefer Google Colab for its GPU options and speed in spinning up a session). I'll be coding in Google Colab.
In Python, we can represent tensors as NumPy arrays, or TensorFlow/PyTorch tensors. I'll show examples for each case.
For ease, you can check out my Colab:
﻿
Let's get our imports out of the way.
import numpy as np
import tensorflow as tf
Below, we'll show you some examples of constructing NumPy arrays and a few operations and handy functions. Credit to Jason Brownlee for the examples in his awesome article here!
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
﻿
>>> a + b
array([5, 7, 9])
﻿
>>> a - b
array([-3, -3, -3])
﻿
>>> a / b
array([0.25, 0.4 , 0.5 ])
﻿
>>> a * b
array([ 4, 10, 18])
﻿
>>> print(a.shape)
(3,)
>>> print(a.dtype)
int64
There's a lot more functionality than what's shown here! We highly recommend you check out the Colab Notebook for more. The gist here is that NumPy arrays/tensors do arithmetic element-wise. They can be ragged and can take on a range of other unique specifications. 
Below, you'll see a few examples of TensorFlow tensors. Note, these are very similar to NumPy arrays. There is a high chance TensorFlow has a TensorFlow equivalent of any NumPy function.
>>> e = np.array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])
>>> e = e.reshape(-1)
>>> e
array([1, 2, 3, 4, 5, 6, 7, 8, 9])
﻿
>>> tf.convert_to_tensor(e)
<tf.Tensor: shape=(9,), dtype=int64, numpy=array([1, 2, 3, 4, 5, 6, 7, 8, 9])>
In addition to ragged tensors, TensorFlow also has sparse tensors and string tensors (reference).﻿﻿
>>> sparse_tensor = tf.sparse.SparseTensor(indices=[[0, 0], [1, 2]],
                                       values=[1, 2],
                                       dense_shape=[3, 4])
﻿
>>> print(sparse_tensor, "\n")
SparseTensor(indices=tf.Tensor(
[[0 0]
 [1 2]], shape=(2, 2), dtype=int64), values=tf.Tensor([1 2], shape=(2,), dtype=int32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))
﻿
>>> print(tf.sparse.to_dense(sparse_tensor))
tf.Tensor(
[[1 0 0 0]
 [0 0 2 0]
 [0 0 0 0]], shape=(3, 4), dtype=int32)
If you haven't already, go check out the Colab Notebook! It takes a little bit of a deeper dive. If you still aren't satisfied, I recommend visiting the documentation page for NumPy or TensorFlow or both. They have amazing guides! In addition to that, you can always do a quick search for online resources/courses. Hopefully from this preview and the examples in the notebook, you learned how tensors are defined and what type of operations are possible with them. Next, we will take a look at how to use tensors with Weights & Biases (W&B).
Using Weights & Biases With TensorsWeights & Biases is a great tool for collaboration and monitoring and tracking things. Whether your tensors are for ML or not hardly matters. Most of your tensor use cases will be in different forms of media, whether that be video, image, constant values, plots or something else. Let's run through a brief example of how to set up logging, then showcase an ML example.
Say you run a virtual physics simulation, and you want to keep track of something like the stress tensor I mentioned earlier over a period of time. How do we do that? Simple! As long as your code uses Python, integrating W&B is as simple as 3-4 lines of code!
# !pip install wandb
import wandb
wandb.login()  # Make sure you have an account. This will prompt you for your API token.
﻿
run = wandb.init(project="My physics project", name="run_1")
﻿
# Code to run simulation here.
# Imagine periodically, a function here returns tensor output describing the physics simulation.
x, y, z = func1()  # Let's say your function returns position in 3 coordinates.   
﻿
# Every log is a time step in the line plot. 
# For instance, if you had a for loop that iterated 10 times, each time a log was made, the line plots on 
# W&B would have exactly 10 data points for an "x", "y", and a "z" graph!
run.log({
	"x": x,
	"y": y,
	"z": z
})
﻿
# Let's say you don't want separate line charts for each dimension. You can do a 3D point cloud instead!
# Below is a trivial example. Imagine you have an array "positions" of shape (n, 3) where n is the number of points 
# (one for each time step) and 3 is for the number of axes (x, y, and z).
positions = np.array([[x+i, y+i, z+i] for i in range(100)])
﻿
run.log({
	"point_scene": wandb.Object3D(positions)
})
﻿
run.finish()
As you can see, logging and visualizing tensors is nothing difficult in W&B. There are a number of other objects that you can log like video, image, table, histograms, etc. Yet this is only a small needle in the haystack of features W&B provides. For more information on what you can log, check here.
W&B has a selection of tools to aid in your tensor-driven endeavors! I covered the tip of the Dashboard iceberg. W&B also allows for saving tensors (or, more generally, data of any kind) as artifacts. These can be paired with W&B Reports to produce effective documents for collaboration.
Figure 4. W&B modularized tools. Source: github/wandb﻿
Let's check out some examples of using W&B for our ML tensor purposes. I'll highlight a recent small project of mine where I set up an image segmentation model baseline with 5-fold cross-validation for Kaggle's HuBMAP + HPA - Hacking the Human Body Competition. Don't worry if these terms are confusing! The point here is to show W&B's flexibility. Here is the project.
Figure 5. My example W&B charts dashboard.
Given an image, the competitor's responsibility is to build an AI system to segment certain parts of the organ within the image. Here I log metric values across time. The code needed to generate these interactive plots is shown below.
run.log({
	"mean_loss": np.mean(train_losses),
	"sum_loss": np.sum(train_losses),
	"val_mean_loss": np.mean(val_losses),
	"val_sum_loss": np.sum(val_losses),
	"val_mean_dice_coeff": np.mean(dice_coeffs),
	"lr": lr
})
That's it! This simple configuration within my Python script generated this range of interactive line plots for tracking and monitoring model performance.
Figure 6. My example W&B tables dashboard.
I constructed a W&B Table object to log my model's worst predictions. I logged onto this table 4 categories: a metric float value, an image of the input, an image of my AI's prediction, and the ground truth corresponding to that input image. How are these images stored? With tensors!
When integrating W&B into your machine learning workflow, you'll often find yourself logging all types of media. Most, if not all of this media is organized into a tensor format.
And again, here is the W&B project for reference. For more information, check out W&B's documentation page here.
Examples Of Tensors In ActionThe following W&B reports cover a range of different topics in ML. They showcase how tensors can be integrated into W&B workflows. 
The Science of Debugging with Weights & Biases Reports
In this article, we look at how Latent Space uses Weights & Biases Reports to identify issues and debug models quickly. 
Interpretability in Deep Learning With Weights & Biases: CAM and Grad-CAM
This article reviews how Grad-CAM counters the common criticism that neural networks are not interpretable.
Adversarial Policies in Multi-Agent Settings
This article explores a range of adversarial examples in multi-agent settings, demonstrating how adversarial policies can win reliably without even playing the game.
Understanding the Effectivity of Ensembles in Deep Learning
In this article, we dissect ensembles in order to provide different insights that are useful for understanding the dynamics of deep neural networks in general.
﻿
ConclusionIn this report, we stepped through the definition of a tensor. We briefly touched on its mathematical background and mainly focused on its ML aspect. We worked through some code examples and showed how to work with tensors in W&B. Lastly, I included some example reports (above) and recommended readings (below) for extra context on tensors. I hope this article taught you as much as it has taught me. Thank you!
Recommended Reading﻿TensorFlow's Guide to Tensors﻿﻿﻿
﻿Jason Brownlee's Gentle Intro to Tensors for ML with NumPy﻿﻿﻿
﻿Ruihang Du's Intuitive Intro to Tensors﻿
﻿