Skip to main content

Advanced Tensorboard Features: Graph Dashboard

An introduction to exploring Computation Graphs of our Machine Learning workflows with Tensorboard
Created on March 16|Last edited on November 11

Introduction

TensorBoard is a tool that provides the measurements and visualizations needed during the machine learning workflow. It enables us to track experiment metrics, visualize computation graphs, debug our code, profile the performance of our model, and much more via a very nice web interface.
In this article, we'll take a look at how some of the more advanced features of Tensorboard (such as computation graph exploration and debugging) can be effectively used in our machine learning workflow in tandem with W&B.



Using Tensorboard with W&B

First, a note about compatibility:
  • Weights & Biases supports patching with Tensorboard to automatically log all the metrics from our machine learning workflow into rich and interactive dashboards.
  • W&B also supports automatic Tensorboard logging with all versions of Tensorflow.
  • For machine learning workflows using PyTorch, Tensorboard logging on Weights & Biases dashboards is supported via TensorBoardX and Tensorboard versions newer than 1.14.
Now, in order to enable automatic logging of Tensorboard with W&B, all we need to do is to set the sync_tensorboard parameter to True while initializing a W&B job.
import wandb
# Just start a W&B run, passing `sync_tensorboard=True)`, to plot your Tensorboard files
wandb.init(project='my-project', sync_tensorboard=True)

# Your Keras, Tensorflow or PyTorch training code using TensorBoard
...

# If running in a notebook, finish the wandb run to upload the tensorboard logs to W&B
wandb.finish()



Exploring Computation Graphs with Tensorboard

If you're writing your machine learning workflow in Tensorflow, the Graph Dashboard in TensorBoard is a really powerful tool for examining not only your model but your entire workflow as a computation graph. Let's dig in:

Exploring Model Architecture

The Tensorboard Graph Dashboard enables us to quickly view a Conceptual Graph of our model’s architecture and ensure it matches our intended design. In the case of a standard Tensorflow workflow, this is just a conceptual view of the Keras model. We can get this view simply by changing the tag from "Default" to "keras" in the Tensorboard Interface.


We can use the Conceptual View to visualize PyTorch Models as well.



Exploring Operation-Level Graph

Tensorboard lets us break down the structure of our entire execution workflow (including our model) in the form of an operation-level graph that shows us the detail of each and every operation.
Effectively, that means our workflow is presented in the form of an interactive interface. Using the op-level graph, we can basically "peel" down abstractions and visualize the lowest level operations which can give us import insights regarding the design and optimization of each and every component of our workflow. Like so:
Using the op-level graph to "peel" the abstractions of the Keras sequential model and taking a peek inside the operations that make up individual Keras layers
Note that the graph is inverted, i.e, data flows from bottom to top, so it’s upside down compared to the code.
💡

Visualizing Custom Tensorflow Operations

So far, we have seen how we can use Tensorboard's Graph Dashboard to visualize models written in Keras and PyTorch. However, it can also help us in a situation where we need to use a custom set of python computations. Assuming that we are using Tensorflow, in such a case, we would want to "autograph" our custom computations tf.function. In such a scenario, we can use the TensorFlow Summary Trace API to log autographed functions for visualization on the TensorBoard Graph Dashboard.
Let us try to visualize the function defined in the following coder snippet using Tensorboard:
# The function to be traced.
@tf.function
def sample_function(x, y):
# A simple hand-rolled layer.
return tf.nn.relu(tf.matmul(x, y))

# Prepare our logger
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
logdir = 'logs/func/%s' % stamp
writer = tf.summary.create_file_writer(logdir)

# Sample data for the function.
x = tf.random.uniform((3, 3))
y = tf.random.uniform((3, 3))

# Bracket the function call with tf.summary.trace_on() and tf.summary.trace_export().
tf.summary.trace_on(graph=True, profiler=True)
# Call only one tf.function when tracing.
z = sample_function(x, y)
with writer.as_default():
tf.summary.trace_export(
name="sample_function_trace", step=0, profiler_outdir=logdir
)
Using Tensorboard to visualize the aforementioned "autographed" function


Examining TPU Compatibility

The Graph Dashboard includes a compatibility checker module that checks for and displays operations that can potentially cause issues when a model is run on a TPU. In order to view a model's TPU compatibility graph, all we need to do is select the TPU Compatibility option on the Graph Dashboard.
The graph below presents the compatible (valid) operations in green and the incompatible (invalid) operations in red. The compatibility summary panel displayed to the right of the graph shows the percentage of all Cloud TPU-compatible operations, their attributes, and a list of incompatible operations for a selected node.
Op-level graph of our Tensorflow workflow with respect to TPU Compatibility
Note that compatibility checker does not assess any operations that are explicitly assigned to a non-TPU device using manual device placement. In addition, the checker does not actually compile the model for execution, so be sure to interpret the results as an estimate of compatibility.
💡




Conclusion

In this report we learned the following:
  • Using Tensorboard with Weights and Biases.
  • Exploring model architecture using the Tensorboard Graph Dashboard for both Tensorflow and PyTorch.
  • Exploring operation level graphs using the Tensorboard Graph Dashboard.
  • Visualizing custom Tensorflow operations using the Tensorboard Graph Dashboard.
  • Examining TPU-compatibility of our codebase using the Tensorboard Graph Dashboard.



Similar Reports


Iterate on AI agents and models faster. Try Weights & Biases today.