LlamaIndex OCR-powered document summarization in banking with W&B Weave

Introduction and challenges

A framework for building GenAI solutions in banking

Exploration: Prompt engineering and model selection in a sandbox.
Prototyping: Building an OCR + RAG pipeline and evaluating it on sample data.
Iteration: Refining the system through feedback, systematic evaluation, and validation.
Production: Deploying the solution with ongoing monitoring and periodic re-validation.

Loan document ingestion: Parsing borrowers’ submitted PDFs (bank statements, pay stubs, etc.) and summarizing key risk factors for credit underwriting.
KYC onboarding automation: Extracting and validating information from ID documents and corporate registries, with an audit trail to satisfy Know-Your-Customer regulations.
Trade confirmation summarization: Reading lengthy trade confirm documents or term sheets and generating concise summaries for traders or compliance reviewers.
Regulatory compliance analytics: Scanning regulatory filings or policies and answering queries (with citations) for risk and compliance teams.
Contract review and abstraction: Leveraging OCR on legal contracts or prospectuses and highlighting important clauses or deviations for legal/risk approval.

Technical components and architecture

W&B Weave: An end-to-end LLMOps platform supporting experiment tracking, dataset versioning, prompt playgrounds, pipeline tracing, evaluation, and production monitoring in one system.
OCR: LlamaIndex Cloud → text extraction. This ingests scanned or PDF documents and outputs machine-readable text.
LLM(s): One or more large language models that will generate answers or summaries. In this Solution Accelerator, we're using OpenAI, but you may refactor this to use another LLM provider or your own fine-tuned model.
Embeddings: OpenAI text-embedding-3-small (1536D)
Generation: GPT-4o
Vector Database: A semantic index to enable the retrieval of relevant text chunks from documents (for RAG). This could be an open-source pgvector/Postgres or a managed service like Pinecone, depending on scalability and hosting requirements. In this Solution Accelerator we use Pinecone.
Orchestration Framework: The OCR step uses LlamaIndex's LlamaExtract component, which integrates seamlessly into programmatic workflows. The extractor outputs strongly-typed Pydantic objects, ensuring each pipeline step reliably passes validated data to the next—critical for financial statements where accuracy and traceability are non-negotiable.

Deeper dive into our OCR LlamaIndex

Understanding parse vs. extract in LlamaIndex

Designing an agent for extraction

Schema-aware: The agent is created with your Pydantic model (like IncomeStatement), so it knows to extract company names as strings and revenue figures as floats
Configurable behavior: Settings like use_reasoning, cite_sources, and extraction_mode define how the agent processes documents
Reusable: Once created with a name (e.g., "income-statement-parser"), the agent can be retrieved and reused across multiple extraction runs, ensuring consistency

Code implementation walkthrough

1. Defining the data schema

class IncomeStatement(BaseModel):
    company_name: str = Field(description="The name of the company")
    total_revenue: float = Field(description="Total revenue amount")
    cost_of_goods_sold: float = Field(description="Cost of goods sold amount")
    gross_profit: float = Field(description="Gross profit amount")
    payroll_expense: float = Field(description="Payroll expense amount")
    depreciation_expense: float = Field(description="Depreciation expense amount")
    total_operating_expenses: float = Field(description="Total operating expenses amount")
    interest_expense: float = Field(description="Interest expense amount")
    taxes: float = Field(description="Taxes amount")
    net_profit: float = Field(description="Net profit or net income amount")

2. Creating and retrieving the extraction agent

@weave.op()
def get_extraction_agent(
    agent_name: str = "income-statement-parser",
    data_schema: BaseModel = IncomeStatement,
) -> Any:
    extractor = LlamaExtract()
    try:
        # Check if an agent with this name already exists
        agent = extractor.get_agent(name=agent_name)
        return agent
    except Exception:
        # If not, create a new one
        config = ExtractConfig(
            use_reasoning=True,
            cite_sources=True,
            extraction_mode=ExtractMode.MULTIMODAL,
        )
        agent = extractor.create_agent(
            name=agent_name, data_schema=data_schema, config=config
        )
        return agent

Key configuration choices:

use_reasoning=True: Enables the agent to explain why it extracted specific values, critical for auditing and validation.
cite_sources=True: Links extracted data back to specific document locations, enabling auditors to verify each number.
extraction_mode=ExtractMode.MULTIMODAL: Handles complex financial documents with tables, charts, and mixed layouts.
Note: Multimodal and Premium are the only configuration choices that allow you to select an LLM for parse or extract.
@weave.op() decorator: Automatically logs this function's execution to Weave, creating a traceable record of which agent configuration was used.

UI-based extraction workflow



Key configuration options in the UI

Extraction mode: Choose between FAST (simpler documents, no OCR), BALANCED (default), MULTIMODAL (visually rich documents with text, tables, and images), or PREMIUM (highest accuracy with OCR and complex table detection)
Extraction target: Select whether to extract from the entire document at once (single result), from each page separately (list of results), or from each Table Row (for structured data extraction)
Extensions for financial documents: Enable Cite Sources to trace extracted data back to specific pages and text, Use Reasoning to understand why the AI extracted specific information, and Confidence Score (beta, MULTIMODAL/PREMIUM only) for quantitative confidence measures
Advanced options: Add an optional System Prompt for custom extraction instructions, specify a Page Range to extract from specific sections (e.g., "1,3,5-7"), enable High Resolution Mode for better OCR on small text, and choose Chunk Mode (Page or Section) for processing large documents
Schema creation: Upload a sample financial document or provide a natural language description (like "Extract revenue, expenses, and profit data from income statements") to Auto-Generate a schema, or Create Manually for full control over field definitions and data types.





Exploration



Prototyping

The embedding search query and which document chunks were retrieved from the ocr-mrm-db index (with their similarity scores)
The retrieved context showing the specific chunks that contained the financial information about Boston Tea Enterprises
The complete prompt sent to GPT-4o, including the retrieved context and the original question
The LLM's response generating the answer: "The net profit for Boston Tea Enterprises, Inc. is $45,000"
All configuration parameters including the chunk size (500), chunk overlap (100), retrieval top_k (5), and temperature (0.7)



import weave
from src.llamaindex.extractor import extract_documents
from src.rag.chunker import chunk_text_with_overlap
from src.rag.embed import create_embeddings
from src.weave.model import RagModel

# Initialize Weave
weave.init('solution-accelerator-mrm-eval')

@weave.op()
def process_document(document_path: str):
    # 1. Extract structured data from document
    print(f"Extracting data from {document_path}...")
    results = extract_documents([document_path])
    extracted_data = results[0]['data']
    print("Data extracted successfully.")
    
    # 2. Convert to text and chunk
    print("Creating text chunks...")
    text_parts = [f"{key.replace('_', ' ').title()}: {value}" 
                  for key, value in extracted_data.items() if value]
    extracted_text = "\n".join(text_parts)
    chunks = chunk_text_with_overlap(extracted_text, chunk_size=500, overlap=100)
    print(f"Created {len(chunks)} chunks.")
    
    # 3. Create embeddings
    print("Creating embeddings...")
    texts_to_embed = [chunk['text'] for chunk in chunks]
    embeddings = create_embeddings(texts_to_embed, model="text-embedding-3-small")
    print(f"Generated {len(embeddings)} embeddings.")
    
    # 4. Upsert to Pinecone
    print("Upserting to Pinecone...")
    model = RagModel(index_name="ocr-mrm-db", namespace="default")
    
    vectors_to_upsert = []
    for i, chunk in enumerate(chunks):
        vectors_to_upsert.append({
            "id": f"doc_{document_path}_{i}",
            "values": embeddings[i],
            "metadata": {"text": chunk['text'], "chunk_number": i}
        })
    
    model.retriever.vector_store.upsert(vectors_to_upsert)
    print(f"Successfully uploaded {len(vectors_to_upsert)} vectors to Pinecone.")

Iteration

Retrieval
Context Recall / Recall@k: how many important docs did we retrieve (k = #retrieved_chunks)
Mean Reciprocal Rank / MRR@k: ranking scorer, how high does the first relevant doc appear (k = #retrieved_chunks)
Generation
Faithfulness: generally whether the answer is grounded in the context (using one of our own pre-built scorers HallucinationFreeScorer)
Numeric consistency: specific regex check whether the numbers actually appeared in the context

1. Create scorers as a function

@weave.op()
def recall_at_k(retrieved_docs: List[str], relevant_docs: List[str], k: Optional[int] = None) -> float:
    """Calculate Recall@k - fraction of relevant documents retrieved in top-k results."""
    if not relevant_docs:
        return 1.0  # No relevant docs to retrieve
    
    if not retrieved_docs:
        return 0.0  # No docs retrieved
    
    # Consider only top-k retrieved documents
    top_k_retrieved = retrieved_docs[:k] if k is not None else retrieved_docs
    
    # Count how many relevant docs are in the top-k retrieved
    retrieved_relevant = 0
    for doc in top_k_retrieved:
        if any(relevant_doc in doc or doc in relevant_doc for relevant_doc in relevant_docs):
            retrieved_relevant += 1
    
    return retrieved_relevant / len(relevant_docs)

2. Create and/or use a Dataset version

# Create and publish Weave dataset
    dataset = weave.Dataset(
        name="financial_rag_eval_synthetic",
        rows=dataset_rows
    )
    dataset_ref = weave.publish(dataset)

# Retrieve and use a published version of a dataset
dataset = weave.ref("reference_here").get()

3. Create an Evaluation, passing through our dataset and scorers. Then we run it

# Create evaluation with all scorers
    evaluation = Evaluation(
        dataset=dataset,
        scorers=[
            recall_at_k_scorer,
            mrr_at_k_scorer, 
            numeric_consistency_scorer,
            simple_faithfulness_scorer
        ],
        preprocess_model_input=process_query,
    )
    
# Run evaluation
evaluation_results = await evaluation.evaluate(model)

Changing the underlying model
Experimenting with how many documents or chunks we choose to retrieve
Changing system prompts

Prompt modifications: Even minor wording changes ("summarize the revenue" vs. "extract total revenue") can significantly alter extraction accuracy or introduce biases
Model swaps: Switching from GPT-4 to Claude or upgrading model versions can change reasoning patterns, hallucination rates, and output reliability
Schema updates: Adding or modifying Pydantic field definitions affects what data gets extracted and validated
Temperature/parameter tuning: Adjusting sampling parameters impacts consistency and creativity in outputs
Tool/function changes: Modifying which tools the agent can access or how they're defined changes decision-making paths

Comparison overview



Quantitative comparison



Qualitative comparison



Production

Production monitoring

Code snippets to capture and add feedback

Create a Dataset from Traces

Conclusion

Loan document ingestion challenge: extracting key credit metrics from varied applicant documents while ensuring no errors in data or breaches of privacy. Weave solution: Full traceability of each extracted field and summary. Auditors can see which source page and value led to a credit ratio. Automated evaluations flag if an output references a data point not found in the documents (preventing hallucinated financials). PII guardrails ensure that, say, a Social Security Number in a pay stub is not echoed back in the summary.
KYC onboarding challenge: verifying ID documents and compliance lists with an audit trail for regulators. Weave solution: Weave Traces log each step in the verification agent (OCR of ID, checks against sanction lists, LLM explanations) with timestamps. This provides evidence of due diligence. Monitors track false negative rates; if the AI misses a known risk keyword, it’s caught in evaluation and can trigger retraining before any regulatory slip-up.
Trade confirmation summarization challenge: distilling complex trade documents into summaries without misrepresenting terms. Weave solution: The retrieval-augmented generation is logged such that every sentence in the summary can be linked to the source text via Weave’s context mapping. Hallucination scorers enforce that nothing beyond the document is introduced. In production, any unusual content (e.g. a number that doesn’t match the source) sets off an alert for a manual review, aligning with traders’ need for accuracy.
Regulatory compliance Q&A challenge: answering questions about regulations by referencing the law text, with complete accuracy. Weave solution: The system uses RAG to quote actual rule text. Weave Evaluations include a correctness judge scorer (an LLM configured to verify if answers are supported by the reg text) to ensure no advice given is incorrect. A citation coverage metric ensures the answer always cites the section of the regulation. This gives compliance officers confidence in using the AI assistant, knowing it won’t fabricate rules.
Contract review and abstraction challenge: reading lengthy contracts and pulling out important clauses while maintaining legal accuracy. Weave solution: During development, different prompt strategies (bullet list vs. paragraph summary) are logged, evaluated and compared to choose the most faithful format. In production, any low-confidence summaries (perhaps measured by an embedding similarity scorer between the summary and original text) can be routed for human lawyer review with Weave capturing those cases for later model improvement.

Getting started