Skip to main content
W&B Fully Connected
>
Articles
>
LLM

Tutorial: Running inference with DeepSeek V3.1 using W&B Inference

Getting set up and running DeepSeek-V3.1, DeepSeek's advanced long-context language model, in Python using W&B Inference. We'll be working with the DeepSeek model.
Brett Young
Created on August 22|Last edited on August 22
Running inference with DeepSeek-V3.1 through W&B Inference powered by CoreWeave is surprisingly straightforward, and will offer you a lot of flexibility in what you can do with it beyond what you might encounter through other interfaces. In this tutorial, we'll have you running inference and exploring advanced capabilities in depth.
Whether you're running inference on massive documents, building multilingual applications, or tackling complex reasoning tasks, this guide provides everything you need to run inference with DeepSeek-V3.1 effectively.

Table of contents

What is DeepSeek-V3.1?W&B WeaveTutorial: Running inference with DeepSeek-V3.1 using W&B InferencePrerequisitesStep 1: Installation & setupStep 2: Environment configurationStep 3: Running basic inference with DeepSeek V3.1Step 4: Advanced DeepSeek V3.1 inference configurationStep 5: Running inference with DeepSeek V3.1's unique capabilitiesMonitoring DeepSeek V3.1 inference with W&B WeaveBest PracticesNext steps


What is DeepSeek-V3.1?

DeepSeek V3.1is a state-of-the-art large language model developed by DeepSeek, explicitly engineered for tasks requiring deep understanding and extensive context processing. What sets DeepSeek V3.1 apart from other language models:
🔍 Extended context window: Supports up to 128,000 tokens in a single run, making it suitable for large codebases, full-length documents, research archives, or extended multi-turn conversations.
🌐 Multilingual excellence: Superior performance across languages with particular strength in Chinese and English, making it ideal for international applications and cross-cultural content analysis.
💻 Code understanding: Sophisticated comprehension and generation capabilities across programming languages, with deep understanding of software architecture and patterns.
🧠 Hybrid reasoning: Offers both thinking and non-thinking modes within one model, allowing faster inference when needed and deeper logical reasoning for complex tasks.

For detailed technical specifications and performance benchmarks, visit the DeepSeek V3.1 model documentation.

W&B Weave

W&B Weave simplifies the process of tracking and analyzing model outputs in your project. To get started with Weave, you'll first import it and initialize it with your project name.
One of its standout features is the @weave.op decorator. In Python, a decorator is a powerful tool that extends the behavior of a function. By placing @weave.op above any function in your code, you're telling Weave to log that function's inputs and outputs automatically. This makes it incredibly easy to keep track of what data goes in and what comes out.
Once your code runs, these logs appear in the Weave dashboard, where you'll find detailed visualizations and traces of the function calls. This not only aids in debugging but also helps structure your experimental data, making it much easier to develop and fine-tune models like DeepSeek V3.1.

Tutorial: Running inference with DeepSeek-V3.1 using W&B Inference

Let's jump right in. This tutorial assumes you're working in a Jupyter notebook, but of course, the code will work in other applications.
We're going to be running inference with the deepseek-ai/DeepSeek-V3.1 model specifically.
If you're not familiar with Jupyter Notebooks, you can get set up in about 5 minutes. I walk you through it in this tutorial.
💡

Prerequisites

Before starting, ensure you have:
  • A Weights & Biases account (you can sign up free here)
  • Python 3.7 or higher installed
  • Basic familiarity with Python and API usage
  • Understanding of your use case requirements (document analysis, code review, multilingual tasks, etc.)

Step 1: Installation & setup

1. Install required packages

To get started running inference with DeepSeek V3.1, all you need to install is OpenAI and Weave. However, we'll also show you how to simplify reviewing multiple outputs using W&B Weave, which makes the process much more efficient.
The code to do this is:
pip install openai wandb weave
Run this in your terminal or Jupyter cell after entering this code.
When you execute the cell, you'll notice an asterisk ([*]) appear between the brackets [ ]. This indicates that the cell is running, and you'll need to wait until the asterisk turns into a number before proceeding.

2. Get your W&B API key

  1. Visit https://wandb.ai/authorize
  2. Copy your API key
  3. Keep it handy for the next step

Step 2: Environment configuration

Setting up your environment variables is crucial for secure and seamless operation. You'll need both your W&B API key.

Option 1: In a Jupyter Notebook

# Set environment variables in your notebook
%env WANDB_API_KEY=your-wandb-api-key-here

Option 2: In Terminal/Shell

export WANDB_API_KEY="your-wandb-api-key-here"

Option 3: In Python script

import os
# Set environment variables programmatically
os.environ["WANDB_API_KEY] = "your-wandb-api-key-here"

Step 3: Running basic inference with DeepSeek V3.1

Hopefully, this hasn't been too painful because now we're at the fun part.
Here's a complete example to start running inference with DeepSeek V3.1:
import os
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant specialized in clear explanations."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    temperature=0.7,
    max_tokens=1000,
)

print(resp.choices[0].message.content)
You'll find the inputs and outputs recorded to your Weave dashboard with the parameters automatically included:


Step 4: Advanced DeepSeek V3.1 inference configuration

Understanding inference parameters

Adjust DeepSeek V3.1's response behavior with these key inference parameters (feel free to play around with them and compare the outputs in Weave!).
import os
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }    
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are a creative writing assistant."},
        {"role": "user", "content": "Write a short story about time travel."}
    ],
    temperature=0.8,
    top_p=0.9,
    max_tokens=2000,
)
print(resp.choices[0].message.content)

Parameter Guidelines:
  • Temperature: Use 0.1-0.3 for analytical tasks, 0.7-0.9 for creative work
  • Top_p: Combine with temperature; 0.9 works well for most applications
This gives us added flexibility to influence our model output. These parameters are also automatically logged to W&B Weave for observability:


Streaming inference responses

For real-time output and better user experience:
import os
import sys
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)

stream = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a comprehensive story about space exploration."}
    ],
    stream=True,
    temperature=0.7,
)

sys.stdout.write("Response: ")
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        sys.stdout.write(delta.content)
        sys.stdout.flush()
print()

We got a streaming response:

With the metrics logged to Weave:

As well as the full output:


Step 5: Running inference with DeepSeek V3.1's unique capabilities

This is where running inference with DeepSeek V3.1 really shines. Let's explore what makes it special.

Long context inference

DeepSeek V3.1 excels at running inference on extensive documents. Here's a practical example:
import os
import io
import requests
import openai
import weave
from pypdf import PdfReader

PROJECT = "wandb_inference"
weave.init(PROJECT)

PDF_URL = "https://docs.aws.amazon.com/pdfs/bedrock-agentcore/latest/devguide/bedrock-agentcore-dg.pdf"
QUESTION = "How does AgentCore memory work?"

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)

r = requests.get(PDF_URL, timeout=60)
r.raise_for_status()

reader = PdfReader(io.BytesIO(r.content))
pages = reader.pages[:100]
text = "\n\n".join(page.extract_text() or "" for page in pages)

doc_snippet = text

prompt = (
    "You analyze AWS Bedrock AgentCore docs and answer using only the provided text. "
    "If something is not in the text, say you cannot find it.\n\n"
    f"Document:\n{doc_snippet}\n\nQuestion: {QUESTION}\n"
    "Cite exact phrases where possible."
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are an expert on AWS Bedrock AgentCore."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.2,
    max_tokens=1500,
)

print(resp.choices[0].message.content)
Which outputs to Weave:


Multilingual inference

Leverage DeepSeek V3.1's multilingual inference capabilities for international development:
import os
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)



code_snippet = """
// Chinese identifiers with English comments
function 计算总价(商品列表, 折扣率) {
    let 总价 = 0;
    for (const 商品 of 商品列表) {
        总价 += 商品.价格 * 商品.数量
    }
    const 折扣金额 = 总价 * 折扣率
    return 总价 - 折扣金额
}

# Python with Chinese docstring
def validate_用户输入(user_data):
    '''
    验证用户输入数据的完整性和有效性
    Validates user input data for completeness and validity
    '''
    required_fields = ['name', 'email', '年龄']
    for field in required_fields:
        if field not in user_data:
            raise ValueError(f"Missing required field: {field}")
    return True
"""

task = (
    "Explain in English what the code does and provide a concise Chinese explanation. "
    "Then suggest improvements for naming consistency and error handling. "
    "Provide a refactored version using one language for identifiers."
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are a senior engineer fluent in English and Chinese."},
        {"role": "user", "content": f"{task}\n\nCode:\n{code_snippet}"}
    ],
    temperature=0.2,
    max_tokens=1200,
)

print(resp.choices[0].message.content)

Which logs to Weave as:


Complex multi-step reasoning inference with DeepSeek V3.1

Utilize DeepSeek V3.1's inference reasoning capabilities for complex problem solving:
import os
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant specialized in SaaS growth strategy."},
        {"role": "user", "content": """
        Our SaaS company is experiencing a 15% monthly churn rate. Our customer acquisition cost (CAC) is $150, 
        average customer lifetime value (CLV) is $800, and monthly recurring revenue per customer is $50. 
        We have 2,000 active customers and are spending $60,000/month on marketing.
        
        Please analyze this situation and provide a comprehensive strategy to improve our metrics, 
        including specific actions, expected timelines, and success metrics.
        """}
    ],
    temperature=0.7,
    max_tokens=1000,
)

print(resp.choices[0].message.content)
Which you'll see in the dashboard:


Monitoring DeepSeek V3.1 inference with W&B Weave

From the final cell you can access the inference output, and copy it from there. If you want to dig deeper into what's going on or review past inference requests (and I recommend you do) you can do so by visiting your Weights & Biases dashboard or clicking the links outputted with the response.
With Weave initialized using your environment variable, all your inference API calls are automatically tracked. Here's what gets logged and how to use it effectively:

What Weave tracks automatically

  • Request details: Model used, parameters, token counts
  • Response data: Content, processing time, success/failure status
  • Usage metrics: Token consumption, API costs, rate limit status
  • Performance: Response latency, throughput patterns

Accessing your logs

  • Visit your W&B project dashboard at: https://wandb.ai/[your-username]/[your-project]
  • Navigate to the "Weave" section
  • View detailed logs, filter by date/model/status
  • Analyze usage patterns and optimize accordingly

Custom Weave annotations

Add custom metadata and organize your API calls:
import os
import openai
import weave

PROJECT = "wandb_inference"
weave.init(PROJECT)

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key=os.getenv("WANDB_API_KEY"),
    project=PROJECT,
    default_headers={
        "OpenAI-Project": "wandb_fc/quickstart_playground"  # replace with your actual team/project
    }
)

@weave.op()
def analyze_customer_feedback(feedback_text, sentiment_threshold=0.5):
    """
    Analyze feedback and return sentiment summary.
    Tracked via weave.op since the OpenAI client has no built-in weave hook.
    """
    resp = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-V3.1",
        messages=[
            {"role": "system", "content": "You score sentiment from -1 to 1 and list key topics."},
            {"role": "user", "content": f"Feedback: {feedback_text}\nThreshold: {sentiment_threshold}"}
        ],
        temperature=0.1,
        max_tokens=500,
    )
    return resp.choices[0].message.content

if __name__ == "__main__":
    out = analyze_customer_feedback(
        "The new update is confusing and slow. I cannot find the features I used daily.",
        sentiment_threshold=0.3,
    )
    print(out)
Which would appear as:


Best Practices

Here are some best practices to follow when testing and/or deploying DeepSeek V3.1 via inference, or any other model for that matter.

Security and Configuration

  • Environment variables: Always store API keys in environment variables, never hardcode them
  • Project organization: Use clear, descriptive project names following the "team/project" format
  • Access control: Limit API key permissions to necessary scopes only

Prompt Engineering for DeepSeek V3.1

  • Leverage long context: Don't hesitate to provide extensive context - DeepSeek V3.1 handles it well
  • Clear instructions: Be specific about the desired output format and style
  • System messages: Use detailed system prompts to establish expertise and context
  • Temperature selection: Lower values (0.1-0.3) for analytical tasks, higher (0.7-0.9) for creative work

Performance Optimization

  • Streaming: Use streaming for longer responses to improve user experience
  • Batch processing: Group similar requests when possible to improve efficiency
  • Token management: Monitor token usage to optimize costs and stay within limits
  • Caching: Implement response caching for frequently requested analyses

Monitoring and Debugging

  • Weave integration: Use Weave's automatic logging for all production calls
  • Custom annotations: Add meaningful metadata to track different use cases
  • Error analysis: Regularly review failed requests to identify patterns
  • Performance tracking: Monitor response times and adjust parameters accordingly

Next steps

Now that you're equipped with comprehensive DeepSeek V3.1 knowledge:
🔗 Explore Advanced Features
📊 Optimize Your Workflow
  • Set up automated monitoring dashboards for your specific use cases
  • Implement A/B testing between different prompting strategies
  • Create custom evaluation metrics for your domain-specific tasks
🚀 Scale Your Implementation
  • Build production pipelines with proper error handling and monitoring
  • Implement cost optimization strategies based on usage patterns
  • Explore integration with other W&B tools for end-to-end ML workflows
📚 Dive Deeper into DeepSeek V3.1
  • Visit the DeepSeek V3.1 model card for detailed capability information
  • Explore community examples and use cases
  • Stay updated with model improvements and new features
With this comprehensive setup, you're ready to harness DeepSeek V3.1's advanced capabilities while maintaining professional-grade monitoring, logging, and error handling through W&B Inference and Weave.
Tags: Articles, LLM, Community Posts

Created with ❤️ on Weights & Biases.

https://wandb.ai/wandb_fc/genai-research/reports/Tutorial-Running-inference-with-DeepSeek-V3-1-using-W-B-Inference--VmlldzoxNDA5ODE5NA
Made with Weights & Biases. Sign up or log in to create reports like this one.
Iterate on AI agents and models faster. Try Weights & Biases today.
Sign upTry W&B now