Tutorial: Fine-tuning OpenAI GPT-OSS

Table of contents



What is OpenAI's GPT-OSS?

How do GPT-OSS models compare to other options?

Leveraging Weights & Biases for observability

Using open-source packages with GPT-OSS considerations

Tutorial: Fine-tuning GPT-OSS-120B

pip install --user --upgrade \
  torch --extra-index-url https://download.pytorch.org/whl/cu128 \
  "transformers>=4.55.0" "trl>=0.20.0" "peft>=0.17.0" \
  datasets trackio wandb weave huggingface_hub ipywidgets "jinja2>=3.1.0"

import wandb

import os

# 0.1 Setup WANDB for training metrics
os.environ["WANDB_PROJECT"] = "YOUR_PROJECT_NAME"  # Change to your desired project!
wandb.init(project=os.environ["WANDB_PROJECT"])

from huggingface_hub import notebook_login

notebook_login()

# 1. LOAD DATASET
from datasets import load_dataset

dataset = load_dataset("HuggingFaceH4/Multilingual-Thinking", split="train")
print(f"Total examples: {len(dataset)}")
print("Sample:", dataset[0])

# 2. LOAD TOKENIZER & PREVIEW TEMPLATE
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")

messages = dataset[0]["messages"]
conversation = tokenizer.apply_chat_template(messages, tokenize=False)
print(conversation)  # Shows how dialogue is formatted

# 3. LOAD BASE MODEL WITH QUANTIZATION
import torch
from transformers import AutoModelForCausalLM, Mxfp4Config

quant_config = Mxfp4Config(dequantize=True)
model_kwargs = dict(
    attn_implementation="eager",
    torch_dtype=torch.bfloat16,
    quantization_config=quant_config,
    use_cache=False,
    device_map="auto",
)
model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", **model_kwargs)

# 4. WRAP WITH LORA (PEFT)
from peft import LoraConfig, get_peft_model

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules="all-linear",
    target_parameters=[
        "7.mlp.experts.gate_up_proj",
        "7.mlp.experts.down_proj",
        "15.mlp.experts.gate_up_proj",
        "15.mlp.experts.down_proj",
        "23.mlp.experts.gate_up_proj",
        "23.mlp.experts.down_proj",
    ],
)
peft_model = get_peft_model(model, peft_config)
peft_model.print_trainable_parameters()

# 5. CONFIGURE TRAINING (& use wandb)
from trl import SFTConfig

training_args = SFTConfig(
    learning_rate=2e-4,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    num_train_epochs=1,
    logging_steps=1,
    max_length=2048,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine_with_min_lr",
    lr_scheduler_kwargs={"min_lr_rate": 0.1},
    output_dir="gpt-oss-20b-multilingual-reasoner",
    report_to="wandb",  # ← use wandb
    push_to_hub=False,  # ← DO NOT push to hub
)

# 6. START TRAINING
from trl import SFTTrainer

trainer = SFTTrainer(
    model=peft_model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

# 7. SAVE MODEL LOCALLY (no pushing to hub)
trainer.save_model(training_args.output_dir)

del trainer
del peft_model
del model

# 8. INFERENCE: LOAD FINE-TUNED MODEL & GENERATE OUTPUT

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import weave; weave.init("gpt-oss")


tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")
model_kwargs = dict(attn_implementation="eager", torch_dtype="auto", use_cache=True, device_map="auto")
base_model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", **model_kwargs).cuda()
peft_model_id = "gpt-oss-20b-multilingual-reasoner"
model = PeftModel.from_pretrained(base_model, peft_model_id)
model = model.merge_and_unload()

# Example generation: User asks in Spanish, wants reasoning in German
import torch

@weave.op
def run_inference(system_prompt, user_prompt, reasoning_language="German"):
    messages = [
        {"role": "system", "content": f"reasoning language: {reasoning_language}"},
        {"role": "user", "content": user_prompt},
    ]
    input_ids = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt",
    ).to(model.device)
    gen_kwargs = {"max_new_tokens": 512, "do_sample": True, "temperature": 0.6}
    output_ids = model.generate(input_ids, **gen_kwargs)
    response = tokenizer.batch_decode(output_ids)[0]
    return response



# Inference with German reasoning
out = run_inference("reasoning language: German", "¿Cuál es el capital de Australia?", "German")
print("\nGerman reasoning output:\n", out)

# Inference with Chinese reasoning (not seen in finetuning)
out_chinese = run_inference("reasoning language: Chinese", "What is the national symbol of Canada?", "Chinese")
print("\nChinese reasoning output:\n", out_chinese)





The power of fine-tuning with GPT-OSS

Domain-Specific Expert Models: You can fine-tune an LLM on domain-specific text (e.g., legal documents, medical journals, finance reports) to create a specialist model. For instance, an open-source model fine-tuned on medical Q&A could become a “MedicalGPT” that provides more accurate and jargon-aware answers in healthcare contexts. Companies often prefer this so that the model speaks their industry’s language and understands niche terminology better than a general model. By using W&B to track training, you can ensure the model is learning the domain concepts (monitor loss on domain-specific validation set) and not diverging.
Customer Service Chatbots: Organizations can fine-tune GPT-OSS models on their customer interaction logs or FAQs to build a chatbot that’s tailored to their products and customers. This might involve training on pairs of customer questions and support answers. The fine-tuned model will learn the company’s style of response and product details. Compared to using a generic model via API, this approach can vastly improve relevance and allow on-premises deployment for data privacy. W&B can be used to evaluate such a model’s performance by tracking metrics like response accuracy or even custom metrics like the percentage of answers containing a required piece of info.
Localized or Translated Models: If you need a model that performs well in a language other than English, fine-tuning a GPT-OSS on data in the target language is a path to get there. For example, one could take an English-trained open model and fine-tune it on a large corpus of text in Spanish, or on a bilingual dataset to make it a translator. The result is a model that is more fluent and accurate in Spanish. Open-source models like BLOOM or XGLM exist for multiple languages, and further fine-tuning can hone them to specific dialects or domains. Using W&B, you could compare the performance of multilingual fine-tuning runs (say, which run yields better perplexity on a Spanish validation set).
Creative Writing and Style Imitation: Fine-tuning can imbue a model with a particular style or creativity. For example, you might fine-tune a model on the works of Shakespeare to get it to produce Shakespearean prose, or on a large collection of sci-fi literature to have it generate science fiction-style text. This is popular in entertainment and game development domains. With W&B, you could keep track of training and generate samples periodically to ensure the model’s style is converging to the desired voice.
Tool-using Agents: A more advanced use case: you can fine-tune models to better perform tasks like code generation or using APIs by training on transcripts of those behaviors. For instance, fine-tuning on a dataset of coding problems and solutions can improve an open-source model’s programming abilities. Projects like SalesForce’s CodeT5 or others have done similar things. If you’re building an assistant that uses tools (search engines, calculators), you could fine-tune a model on dialogues where the model demonstrates those tool usages. This often requires a lot of careful data curation, and W&B’s experiment tracking can help systematically iterate on the fine-tuning (trying different data filtering strategies and comparing model outputs each time).

Do you have enough data to fine-tune without overfitting? If not, consider data augmentation or starting with a model that’s already instruction-tuned.
Is your task better served by a classification-style fine-tuning or generative fine-tuning? (Our example was generative. If you wanted something like sentiment classification, you’d fine-tune in a slightly different way, perhaps using a classification head or prompting style.)
How will you evaluate success? Define and log metrics that reflect the model’s real-world performance for that use (e.g., accuracy, ROUGE, customer satisfaction surveys, etc.).

Maximizing your GPT-OSS fine-tuned model

Exploring reasoning levels and advanced capabilities

Leveraging agentic capabilities in your specialized model

Scaling and deployment considerations

Advanced fine-tuning techniques for GPT-OSS

Monitoring and maintaining model quality

Conclusion