Contact
LOG IN
Sign up
Platform
Weave: AIOps
Inference
Models: MLOps
Core
Solutions
Use Cases
Industries
Case Studies
Enterprise
Resources
Docs
Pricing
Contact LOG IN Sign up

LLM apps: Evaluation

Enroll for free
LLM apps: Evaluation
Develop techniques for building, optimizing, and scaling AI evaluators with minimal human input. Learn to build reliable evaluation pipelines for LLM applications by combining programmatic checks with LLM-based judges.
2 Hours
Free

Learnings & outcomes

At the end of the course you will:

  • Understand key principles, implementation methods, and appropriate use cases for LLM application evaluation
  • Learn how to create a working LLM as a judge
  • Align your auto-evaluations with minimal human input

Curriculum

  • Welcome to the course
  • Evaluation basics
  • Programmatic and LLM Evaluations
  • Alignment
  • Case study: Google: Imagen and Veo
  • Case study: Open Hands
  • Automatic LLM Evaluators
  • Conclusion and course assignment
Enroll for free
In partnership with
In partnership with
Reviews
Extremely Valuable and Well-Structured Course!
This course exceeded my expectations in every way. It provided a perfect balance between theory and hands-on practice, allowing me to deeply understand how to evaluate LLM applications effectively. The real-world examples and step-by-step projects made complex topics like BLEU, ROUGE, F1 Score, and human evaluation strategies feel approachable and actionable. I especially appreciated the focus on building practical workflows using tools like Weights & Biases, and the constant emphasis on scalability and real-world application. Highly recommend this course to anyone serious about working with LLMs!
Stefano Caruso
Good collection of information about LLM Evaluation.
Even though it still have some of the limitation of hard to evaluate the subjective question, it still be a good starting point to have a LLM evaluation system that can automatically evaluate LLM application automatically
ZZJ
Great learning at weights and biases.
It was a great learning opportunity and at the time when AI is at its peak its really helpful to know more about it.
Aditya
Course instructors

Ayush Thakur

Weights & Biases
AI Engineer
Ayush Thakur is an AI Engineer at Weights and Biases and Google Developer Expert in Machine Learning (TensorFlow). He is interested in everything computer vision and representation learning. For the past 2 years he’s been working with LLMs and have covered RLHF and how and what of building LLM-based systems.
Explore our other courses
View all courses
GenAI

AI Engineering: Agents

In this course you'll learn to work with reasoning models to build production-grade AI agents that can autonomously tackle complex workflows, from deep analysis to support agents. Learn to architect and evaluate robust systems that combine LLMs, tools, memory, and planning to ship real-world applications.
Learn more
GenAI

RAG++ : From POC to production

Practical RAG techniques for engineers: learn production-ready solutions from industry experts to optimize performance, cut costs, and enhance the accuracy and relevance of your applications.
Learn more
GenAI

Developer’s guide to LLM prompting

This course covers everything you need to get started with prompt engineering, from system prompts and structural techniques to model-specific strategies. Using a text-understanding use case, you will gain experience with code and industry knowledge.
Learn more
GenAI

LLM engineering: Structured outputs

Improve your LLM engineering skills with Jason Liu, the author of the Instructor library. Learn about structured JSON output handling, function calling, complex validations with Pydantic and more in this short and helpful course!
Learn more
GenAI

Building LLM-powered apps

Learn how to build LLM-powered applications using LLM APIs, Langchain and Weights & Biases LLM tooling. This course will guide you through the entire process of designing, experimenting, and evaluating LLM-based apps.
Learn more
GenAI

Training and fine-tuning LLMs

Learn to harness the power of LLMs with our comprehensive course. Discover the importance and history of LLMs, explore their architecture, training techniques, and fine-tuning methods. Gain hands-on experience with practical recipes from Jonathan Frankle (MosaicML), and other industry leaders,and learn cutting-edge techniques like LoRA and Prefix Tuning. Perfect for machine learning engineers, data scientists, researchers, and NLP enthusiasts. Stay ahead of the curve and become an expert in LLMs.
Learn more
GenAI
W&B PLATFORM

Weights & Biases 101: Weave

This course introduces Weave by Weights & Biases, a toolkit for developing Generative AI applications. Learn to log, debug, and evaluate language model workflows, ensuring accurate and consistent results across your projects.
Learn more