Community Spotlight: Lightning Pod Series
The Lightning Pod Series is a set of public repos built to help anyone who is pairing Lightning AI’s PyTorch Lightning with W&B. Here's what you need to know
Created on October 19|Last edited on October 19
Comment
This report was created in collaboration with Megan Lo of the Weights & Biases community team. Here's the original.
💡
Introduction
Feeling unprepared for the 'real world' is a sentiment many graduates can relate to. And for those who finished their degree during the pandemic— those nerves are heightened even more.
While a lot of people might just enter the workforce and hope for the best, Justin Goheen, creator of Lightning Pod and Lightning Pod Vision, went for a more practical approach. "When I started interviewing for research engineering roles immediately after graduation, it became apparent that a notebooks first approach had left me unprepared for the roles I was interviewing for; so I created the Lightning Pod projects to close the skill gaps that were identified in my interviews" Justin said.
Regarding open-sourcing the projects, Justin shared “I decided to make the repositories public and maintain them because they are useful for other people who are just beginning their learning journey, or who are experienced with research style programming in notebooks, but want to pivot to writing Python libraries”.
What is Lighting Pod?
In the simplest terms, Lightning Pod is a template repository for creating new PyTorch Lightning projects that default to using the W&B integration for experiment management. The project is structured as a Python library, allowing researchers and engineers to clone Lightning Pod, and have immediate access to the tools they need to be successful after installing the project to their virtual environment.
Aside from Lightning, PyTorch, and W&B, Lightning Pod also installs tools to help with command line interfaces, formatting, testing, static type checking, and git hooks—helping fellow engineers and researchers to create professional-looking and maintainable code bases.
Justin's work doesn't stop there. Building on Lightning Pod, Justin’s Lightning Pod Vision gets more granular with W&B and PyTorch Lightning in an applied Vision Transformers project. This repo dives into valuable concepts centered on logging W&B Sweeps as a Lightning App and visualizing results with Plotly Dash.
"The main goal is to help people create something they can show a potential employer as a portfolio project" said Justin. Adding “the base version of Lightning Pod Vision is an excellent starting point; however, individuals should build on the example to show understanding, and improve their fork of the project as desired, so that they can truly make it their own”.
Why Lightning Pod uses Weights & Biases?
It's no secret that W&B can be integrated quickly and easily with an astounding number of repos (9,000 and counting to be exact!). And that convenience factor is precisely why Justin chose W&B for experiment management. All it took to get set up with W&B was just a few lines of code, and the result is a central space to collect all logs and results—providing a detailed system of record of all model training.
On a personal level, Justin decided on W&B as he saw the platform commonly shown during his job search. Many of the companies he wanted to apply to were already using W&B as part of their ML workflow. Justin explains, "It makes sense to learn tools that employers have listed in the job description as a required or preferred skill. I began learning to use W&B exactly for this reason, and kept it in my personal projects because it also happens to be a really nice tool to use and is integrated with Lightning".
What's next for Lightning Pod?
As with many repos, maintenance will be a key focus for Lightning Pod and Lightning Pod Vision. With more releases coming from Lightning, PyTorch, and W&B, it's important to keep pace with these new updates so the project will remain useful for the foreseeable future.
What started as an endeavor to upskill and meet the demands of the competitive job market soon became a project that could help others. The Lightning Pod series provides a stepping stone for building and training models with W&B, Lightning, and PyTorch—all extremely popular tools widely used in the ML space.
How can you get involved with Lightning Pod?
More than contributions, Justin hopes for the community to continue trying out the repos, especially in different environments, and creating new examples for the Lightning and W&B communities. Any feedback is greatly appreciated and will help keep the project useful and relevant.
Justin also encourages W&B and Lightning users to contribute to both tools; as open source contributions also helped him to close skill gaps and eventually land a role at Lightning AI as a Developer Advocate.
Add a comment