Plotting with Pandas and Weights & Biases: Step-by-Step Guide

Load data files in CSV format into a Pandas "DataFrame" format.
Perform operations across columns using a simple interface, ex: df["info"] = df["feat_1"] + df["feat_2"]
Perform queries on your dataframe using simple Python expressions, ex: df.query("info > 100")

Table of Contents

Introduction to Data Plotting and Its Significance in Visualizing Data

Different Types of Plots Available in Pandas for Various Data Scenarios

We can use the .plot() function to create line charts
We can use the "bar" argument for bar charts
We can use the "hist" argument for histograms
We can use the "box" argument for box plots.

Leveraging Weights & Biases for Data Plotting

The wandb.plot.line() function allows us to log a custom line plot and the wandb.plot.line_series() function allows us to plot multi-line plots.
The wandb.plot.scatter() function allows us to log a custom scatter plot
The wandb.plot.bar() function allows us to log a custom bar plot.
The wandb.plot.histogram() function allows us to log a custom histogram.
We can also create interactive Custom Charts by providing data as W&B Tables and then plotting using the wandb.plot_table() function.

Benefits of Using Weights & Biases for Plotting With Pandas

Integration of Weights & Biases With Pandas for Interactive and Collaborative Data Plotting

import wandb
import pandas as pd

# Read our CSV into a new DataFrame
pandas_dataframe = pd.read_csv("data.csv")

# Convert the DataFrame into a W&B Table
wandb_table = wandb.Table(dataframe=pandas_dataframe)

# Add the table to an Artifact to increase the row 
wandb_table_artifact = wandb.Artifact(
    "wandb_artifact", 
    type="dataset")        
wandb_table_artifact.add(wandb_table, "table")

# Log the raw csv file within an artifact to preserve our data
wandb_table_artifact.add_file("data.csv")

# Start a W&B run to log data
run = wandb.init(project="...")

# Log the table to visualize with a run...
run.log({"data": wandb_table})

# and Log as an Artifact
run.log_artifact(wandb_table_artifact)

Best Practices for Effective Data Plotting

Choosing Appropriate Plot Types for Different Data Scenarios

Optimizing Visualization Aesthetics and Readability

Use contrastive colors to highlight the key information!
Make sure that the order in which you plot has an inherent story and leads to a conclusion rather than being a collection of abstract facts about your data.
Make sure that you try and maintain a consistent order and size of your plots.

Iterative Exploration and Refinement Using Weights & Biases



Challenges and Considerations

Handling Large Datasets and Performance Optimization

Try to pre-process your dataset in smaller batches and then merge it into a single dataframe
Consider writing Cython for Pandas
Use the "numba" backend to utilize JIT (Just-in-Time) compilation

Addressing Missing or Inconsistent Data for Accurate Plotting

You can use the .dropna(), method to remove missing values from your dataframe. You can apply this function across rows or columns or a subset of the data.
Your dataframe can contain multiple types of data types, each requiring a different type of specific cleaning technique. If it's numeric features, then ideally, try to normalize them. If there are dates, consider ensuring they are in the same format. If they are categorical, then consider one-hot encoding them.

Conclusion